feat(sprint10): Phase 1 — Data model + bank statement parsers (MT940, CAMT.053, CSV)
Implements the Sprint 10 Phase 1 foundation for the Smart Payment Import feature:
Domain layer:
- 3 new enums: BankFormat (MT940, CAMT053, CSV), ImportSessionStatus, MatchStatus
- StaffPermission.FINANCE_IMPORT
- AuditEventType: BANK_IMPORT_STARTED/COMPLETED/FAILED + BANK_PAYMENT_CONFIRMED
- NotificationType.BANK_IMPORT_COMPLETED
- ConsentType.BANK_DATA (DSGVO consent for IBAN storage)
- 3 new entities: BankImportSession, BankTransaction, CsvColumnMapping
- Member: + iban (VARCHAR 34) + ibanConsentDate
- MemberStatus.LEFT (semantic alias for RESIGNED, referenced by Sprint 9 RetentionService)
Persistence:
- V30__bank_import_sessions.sql
- V31__bank_transactions.sql
- V32__csv_column_mappings.sql (also adds iban + iban_consent_date to members)
- 3 Spring Data repositories
Parser infrastructure (cannamanage-service/src/main/java/de/cannamanage/service/bankimport):
- BankStatementParser interface (Strategy pattern, Spring-injected list)
- ParsedTransaction + ParseResult records
- BankStatementParseException (parse errors)
- Mt940Parser: custom state machine, CENTURY_BOUNDARY=70 for YY→YYYY, proprietary
header tolerance (skips lines before first :20: for StarMoney/WISO/Hibiscus wrappers)
- Camt053Parser: StAX streaming with XXE hardening (IS_SUPPORTING_EXTERNAL_ENTITIES,
SUPPORT_DTD, IS_REPLACING_ENTITY_REFERENCES all false); supports camt.053.001.02
and camt.053.001.08 namespaces
- CsvBankParser: Apache Commons CSV with configurable columns per club; German number
format ("1.234,56"); ISO-8859-1 default encoding
- BankStatementParserService: filename-extension hint + content probe; throws
UnrecognizedFormatException when no parser claims the file
Build verified via Docker (cannamanage-api:sprint10-phase1).
Sprint 9 fix (incidental, required to compile):
- Added MemberStatus.LEFT (Sprint 9 RetentionService referenced it but the enum
value was missing)
- MemberListRegistryGenerator: added LEFT to formatStatus() switch (mapped to
"Ausgetreten", same as RESIGNED)
Sprint 10 docs: analysis, plan, plan-review, testplan.
Co-Authored-By: Lumen <lumen@cannamanage.de>
This commit is contained in:
Patrick Plate
@@ -0,0 +1,25 @@
|
||||
-- Sprint 10: Bank statement import sessions
|
||||
-- Each upload of a bank statement creates one session, which is then matched + reviewed by an admin.
|
||||
-- Status flow: PENDING → IN_REVIEW → COMPLETED (or FAILED at any point).
|
||||
-- Once COMPLETED, the session is immutable per GoBD requirements (§147 AO).
|
||||
CREATE TABLE bank_import_sessions (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
tenant_id UUID NOT NULL,
|
||||
club_id UUID NOT NULL REFERENCES clubs(id) ON DELETE CASCADE,
|
||||
filename VARCHAR(255) NOT NULL,
|
||||
format VARCHAR(20) NOT NULL, -- MT940, CAMT053, CSV
|
||||
total_transactions INTEGER NOT NULL DEFAULT 0,
|
||||
matched_count INTEGER NOT NULL DEFAULT 0,
|
||||
confirmed_count INTEGER NOT NULL DEFAULT 0,
|
||||
skipped_count INTEGER NOT NULL DEFAULT 0,
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'PENDING', -- PENDING, IN_REVIEW, COMPLETED, FAILED
|
||||
uploaded_by UUID NOT NULL REFERENCES users(id),
|
||||
error_message TEXT,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
|
||||
completed_at TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX idx_bank_import_sessions_tenant ON bank_import_sessions(tenant_id);
|
||||
CREATE INDEX idx_bank_import_sessions_club ON bank_import_sessions(club_id);
|
||||
CREATE INDEX idx_bank_import_sessions_status ON bank_import_sessions(club_id, status);
|
||||
CREATE INDEX idx_bank_import_sessions_created ON bank_import_sessions(club_id, created_at DESC);
|
||||
@@ -0,0 +1,32 @@
|
||||
-- Sprint 10: Parsed bank transactions
|
||||
-- One row per transaction in an uploaded bank statement.
|
||||
-- amount_cents: positive = incoming (potential member payment), negative = outgoing (expense).
|
||||
-- match_status drives the review UI: UNMATCHED/SUGGESTED/MATCHED/CONFIRMED/SKIPPED.
|
||||
-- CASCADE on session delete: discarding a draft session also deletes its parsed rows.
|
||||
-- SET NULL on member/payment delete: history is preserved even if the matched entity is removed.
|
||||
CREATE TABLE bank_transactions (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
tenant_id UUID NOT NULL,
|
||||
session_id UUID NOT NULL REFERENCES bank_import_sessions(id) ON DELETE CASCADE,
|
||||
club_id UUID NOT NULL REFERENCES clubs(id) ON DELETE CASCADE,
|
||||
booking_date DATE NOT NULL,
|
||||
value_date DATE,
|
||||
amount_cents INTEGER NOT NULL, -- positive = incoming, negative = outgoing
|
||||
currency VARCHAR(3) NOT NULL DEFAULT 'EUR',
|
||||
reference_text TEXT, -- Verwendungszweck
|
||||
counterparty_name VARCHAR(300),
|
||||
counterparty_iban VARCHAR(34),
|
||||
bank_reference VARCHAR(100),
|
||||
match_status VARCHAR(20) NOT NULL DEFAULT 'UNMATCHED',-- UNMATCHED, SUGGESTED, MATCHED, CONFIRMED, SKIPPED
|
||||
match_confidence INTEGER, -- 0-100, only populated when match_status != UNMATCHED
|
||||
matched_member_id UUID REFERENCES members(id) ON DELETE SET NULL,
|
||||
matched_payment_id UUID REFERENCES payments(id) ON DELETE SET NULL,
|
||||
skip_reason VARCHAR(100),
|
||||
created_at TIMESTAMP NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_bank_transactions_tenant ON bank_transactions(tenant_id);
|
||||
CREATE INDEX idx_bank_transactions_session ON bank_transactions(session_id);
|
||||
CREATE INDEX idx_bank_transactions_club_status ON bank_transactions(club_id, match_status);
|
||||
CREATE INDEX idx_bank_transactions_member ON bank_transactions(matched_member_id);
|
||||
CREATE INDEX idx_bank_transactions_payment ON bank_transactions(matched_payment_id);
|
||||
@@ -0,0 +1,31 @@
|
||||
-- Sprint 10: CSV column mapping templates + member IBAN fields
|
||||
-- CSV files have no standard layout — each bank uses different columns/encodings.
|
||||
-- Admins create a named mapping per bank (e.g. "Sparkasse Export") that the parser reuses.
|
||||
CREATE TABLE csv_column_mappings (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
tenant_id UUID NOT NULL,
|
||||
club_id UUID NOT NULL REFERENCES clubs(id) ON DELETE CASCADE,
|
||||
name VARCHAR(100) NOT NULL, -- e.g. "Sparkasse Export"
|
||||
date_column INTEGER NOT NULL,
|
||||
amount_column INTEGER NOT NULL,
|
||||
reference_column INTEGER,
|
||||
counterparty_column INTEGER,
|
||||
iban_column INTEGER,
|
||||
delimiter VARCHAR(5) NOT NULL DEFAULT ';',
|
||||
date_format VARCHAR(20) NOT NULL DEFAULT 'dd.MM.yyyy',
|
||||
decimal_separator VARCHAR(1) NOT NULL DEFAULT ',',
|
||||
skip_header_rows INTEGER NOT NULL DEFAULT 1,
|
||||
encoding VARCHAR(20) NOT NULL DEFAULT 'ISO-8859-1',
|
||||
is_default BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_csv_column_mappings_tenant ON csv_column_mappings(tenant_id);
|
||||
CREATE INDEX idx_csv_column_mappings_club ON csv_column_mappings(club_id);
|
||||
|
||||
-- Add optional IBAN fields to members.
|
||||
-- Both columns are intentionally NULLABLE — IBAN is only populated after explicit
|
||||
-- BANK_DATA consent (DSGVO Art. 6(1)(a)). ibanConsentDate records when consent was given.
|
||||
-- PostgreSQL adds nullable columns instantly (no table rewrite), safe for production.
|
||||
ALTER TABLE members ADD COLUMN IF NOT EXISTS iban VARCHAR(34);
|
||||
ALTER TABLE members ADD COLUMN IF NOT EXISTS iban_consent_date TIMESTAMP;
|
||||
@@ -0,0 +1,91 @@
|
||||
package de.cannamanage.domain.entity;
|
||||
|
||||
import de.cannamanage.domain.enums.BankFormat;
|
||||
import de.cannamanage.domain.enums.ImportSessionStatus;
|
||||
import jakarta.persistence.*;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Sprint 10 — One upload of a bank statement file. Owns the parsed {@link BankTransaction}s
|
||||
* and tracks review progress until the admin marks the session COMPLETED.
|
||||
* <p>
|
||||
* Status lifecycle: {@code PENDING} → {@code IN_REVIEW} → {@code COMPLETED}.
|
||||
* After COMPLETED the session is immutable per GoBD (§147 AO).
|
||||
* {@code FAILED} is a terminal state for parse errors or discarded sessions.
|
||||
*/
|
||||
@Entity
|
||||
@Table(name = "bank_import_sessions")
|
||||
public class BankImportSession extends AbstractTenantEntity {
|
||||
|
||||
@Column(name = "club_id", nullable = false, updatable = false)
|
||||
private UUID clubId;
|
||||
|
||||
@Column(name = "filename", nullable = false, length = 255)
|
||||
private String filename;
|
||||
|
||||
@Enumerated(EnumType.STRING)
|
||||
@Column(name = "format", nullable = false, length = 20)
|
||||
private BankFormat format;
|
||||
|
||||
@Column(name = "total_transactions", nullable = false)
|
||||
private Integer totalTransactions = 0;
|
||||
|
||||
@Column(name = "matched_count", nullable = false)
|
||||
private Integer matchedCount = 0;
|
||||
|
||||
@Column(name = "confirmed_count", nullable = false)
|
||||
private Integer confirmedCount = 0;
|
||||
|
||||
@Column(name = "skipped_count", nullable = false)
|
||||
private Integer skippedCount = 0;
|
||||
|
||||
@Enumerated(EnumType.STRING)
|
||||
@Column(name = "status", nullable = false, length = 20)
|
||||
private ImportSessionStatus status = ImportSessionStatus.PENDING;
|
||||
|
||||
@Column(name = "uploaded_by", nullable = false)
|
||||
private UUID uploadedBy;
|
||||
|
||||
@Column(name = "error_message", columnDefinition = "TEXT")
|
||||
private String errorMessage;
|
||||
|
||||
@Column(name = "completed_at")
|
||||
private Instant completedAt;
|
||||
|
||||
// Getters and setters
|
||||
|
||||
public UUID getClubId() { return clubId; }
|
||||
public void setClubId(UUID clubId) { this.clubId = clubId; }
|
||||
|
||||
public String getFilename() { return filename; }
|
||||
public void setFilename(String filename) { this.filename = filename; }
|
||||
|
||||
public BankFormat getFormat() { return format; }
|
||||
public void setFormat(BankFormat format) { this.format = format; }
|
||||
|
||||
public Integer getTotalTransactions() { return totalTransactions; }
|
||||
public void setTotalTransactions(Integer totalTransactions) { this.totalTransactions = totalTransactions; }
|
||||
|
||||
public Integer getMatchedCount() { return matchedCount; }
|
||||
public void setMatchedCount(Integer matchedCount) { this.matchedCount = matchedCount; }
|
||||
|
||||
public Integer getConfirmedCount() { return confirmedCount; }
|
||||
public void setConfirmedCount(Integer confirmedCount) { this.confirmedCount = confirmedCount; }
|
||||
|
||||
public Integer getSkippedCount() { return skippedCount; }
|
||||
public void setSkippedCount(Integer skippedCount) { this.skippedCount = skippedCount; }
|
||||
|
||||
public ImportSessionStatus getStatus() { return status; }
|
||||
public void setStatus(ImportSessionStatus status) { this.status = status; }
|
||||
|
||||
public UUID getUploadedBy() { return uploadedBy; }
|
||||
public void setUploadedBy(UUID uploadedBy) { this.uploadedBy = uploadedBy; }
|
||||
|
||||
public String getErrorMessage() { return errorMessage; }
|
||||
public void setErrorMessage(String errorMessage) { this.errorMessage = errorMessage; }
|
||||
|
||||
public Instant getCompletedAt() { return completedAt; }
|
||||
public void setCompletedAt(Instant completedAt) { this.completedAt = completedAt; }
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
package de.cannamanage.domain.entity;
|
||||
|
||||
import de.cannamanage.domain.enums.MatchStatus;
|
||||
import jakarta.persistence.*;
|
||||
|
||||
import java.time.LocalDate;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Sprint 10 — One transaction line parsed from a bank statement file.
|
||||
* <p>
|
||||
* Sign convention for {@code amountCents}: <strong>positive = incoming</strong>
|
||||
* (potential member payment), <strong>negative = outgoing</strong> (expense).
|
||||
* <p>
|
||||
* Match flow: parser writes UNMATCHED → matching engine sets MATCHED/SUGGESTED →
|
||||
* admin sets CONFIRMED or SKIPPED. CONFIRMED links to the created {@link Payment}
|
||||
* via {@code matchedPaymentId}.
|
||||
* <p>
|
||||
* Foreign keys to {@code members} and {@code payments} are not modelled as JPA
|
||||
* relationships to keep the entity flat (UUIDs only) — consistent with the rest
|
||||
* of the schema (see {@link Payment} for the same pattern). Deletes use
|
||||
* {@code SET NULL} so transaction history survives member/payment removal.
|
||||
*/
|
||||
@Entity
|
||||
@Table(name = "bank_transactions")
|
||||
public class BankTransaction extends AbstractTenantEntity {
|
||||
|
||||
@Column(name = "session_id", nullable = false, updatable = false)
|
||||
private UUID sessionId;
|
||||
|
||||
@Column(name = "club_id", nullable = false, updatable = false)
|
||||
private UUID clubId;
|
||||
|
||||
@Column(name = "booking_date", nullable = false)
|
||||
private LocalDate bookingDate;
|
||||
|
||||
@Column(name = "value_date")
|
||||
private LocalDate valueDate;
|
||||
|
||||
/** Positive = incoming, negative = outgoing. Stored in cents to avoid floating-point. */
|
||||
@Column(name = "amount_cents", nullable = false)
|
||||
private Integer amountCents;
|
||||
|
||||
@Column(name = "currency", nullable = false, length = 3)
|
||||
private String currency = "EUR";
|
||||
|
||||
/** German "Verwendungszweck" — free-text payment reference. */
|
||||
@Column(name = "reference_text", columnDefinition = "TEXT")
|
||||
private String referenceText;
|
||||
|
||||
@Column(name = "counterparty_name", length = 300)
|
||||
private String counterpartyName;
|
||||
|
||||
@Column(name = "counterparty_iban", length = 34)
|
||||
private String counterpartyIban;
|
||||
|
||||
/** Bank's own internal transaction reference (EREF, KREF, MREF). */
|
||||
@Column(name = "bank_reference", length = 100)
|
||||
private String bankReference;
|
||||
|
||||
@Enumerated(EnumType.STRING)
|
||||
@Column(name = "match_status", nullable = false, length = 20)
|
||||
private MatchStatus matchStatus = MatchStatus.UNMATCHED;
|
||||
|
||||
/** 0-100; only meaningful when {@link #matchStatus} is not UNMATCHED. */
|
||||
@Column(name = "match_confidence")
|
||||
private Integer matchConfidence;
|
||||
|
||||
@Column(name = "matched_member_id")
|
||||
private UUID matchedMemberId;
|
||||
|
||||
@Column(name = "matched_payment_id")
|
||||
private UUID matchedPaymentId;
|
||||
|
||||
@Column(name = "skip_reason", length = 100)
|
||||
private String skipReason;
|
||||
|
||||
// Getters and setters
|
||||
|
||||
public UUID getSessionId() { return sessionId; }
|
||||
public void setSessionId(UUID sessionId) { this.sessionId = sessionId; }
|
||||
|
||||
public UUID getClubId() { return clubId; }
|
||||
public void setClubId(UUID clubId) { this.clubId = clubId; }
|
||||
|
||||
public LocalDate getBookingDate() { return bookingDate; }
|
||||
public void setBookingDate(LocalDate bookingDate) { this.bookingDate = bookingDate; }
|
||||
|
||||
public LocalDate getValueDate() { return valueDate; }
|
||||
public void setValueDate(LocalDate valueDate) { this.valueDate = valueDate; }
|
||||
|
||||
public Integer getAmountCents() { return amountCents; }
|
||||
public void setAmountCents(Integer amountCents) { this.amountCents = amountCents; }
|
||||
|
||||
public String getCurrency() { return currency; }
|
||||
public void setCurrency(String currency) { this.currency = currency; }
|
||||
|
||||
public String getReferenceText() { return referenceText; }
|
||||
public void setReferenceText(String referenceText) { this.referenceText = referenceText; }
|
||||
|
||||
public String getCounterpartyName() { return counterpartyName; }
|
||||
public void setCounterpartyName(String counterpartyName) { this.counterpartyName = counterpartyName; }
|
||||
|
||||
public String getCounterpartyIban() { return counterpartyIban; }
|
||||
public void setCounterpartyIban(String counterpartyIban) { this.counterpartyIban = counterpartyIban; }
|
||||
|
||||
public String getBankReference() { return bankReference; }
|
||||
public void setBankReference(String bankReference) { this.bankReference = bankReference; }
|
||||
|
||||
public MatchStatus getMatchStatus() { return matchStatus; }
|
||||
public void setMatchStatus(MatchStatus matchStatus) { this.matchStatus = matchStatus; }
|
||||
|
||||
public Integer getMatchConfidence() { return matchConfidence; }
|
||||
public void setMatchConfidence(Integer matchConfidence) { this.matchConfidence = matchConfidence; }
|
||||
|
||||
public UUID getMatchedMemberId() { return matchedMemberId; }
|
||||
public void setMatchedMemberId(UUID matchedMemberId) { this.matchedMemberId = matchedMemberId; }
|
||||
|
||||
public UUID getMatchedPaymentId() { return matchedPaymentId; }
|
||||
public void setMatchedPaymentId(UUID matchedPaymentId) { this.matchedPaymentId = matchedPaymentId; }
|
||||
|
||||
public String getSkipReason() { return skipReason; }
|
||||
public void setSkipReason(String skipReason) { this.skipReason = skipReason; }
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package de.cannamanage.domain.entity;
|
||||
|
||||
import jakarta.persistence.*;
|
||||
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Saved CSV column mapping template for a club.
|
||||
* <p>
|
||||
* CSV bank exports have no standard layout: column order, delimiter, encoding,
|
||||
* date format, and decimal separator all vary by bank. Rather than asking the
|
||||
* admin to re-configure on every upload, mappings are saved per bank
|
||||
* (e.g. "Sparkasse Export", "DKB Online").
|
||||
* <p>
|
||||
* One mapping per club may be flagged as {@link #isDefault} — used to
|
||||
* pre-populate the upload wizard.
|
||||
*/
|
||||
@Entity
|
||||
@Table(name = "csv_column_mappings")
|
||||
public class CsvColumnMapping extends AbstractTenantEntity {
|
||||
|
||||
@Column(name = "club_id", nullable = false, updatable = false)
|
||||
private UUID clubId;
|
||||
|
||||
/** User-facing label, e.g. "Sparkasse Export" or "DKB Online". */
|
||||
@Column(name = "name", nullable = false, length = 100)
|
||||
private String name;
|
||||
|
||||
/** 0-based column index of the booking date. */
|
||||
@Column(name = "date_column", nullable = false)
|
||||
private Integer dateColumn;
|
||||
|
||||
/** 0-based column index of the amount field (sign convention: bank's own). */
|
||||
@Column(name = "amount_column", nullable = false)
|
||||
private Integer amountColumn;
|
||||
|
||||
@Column(name = "reference_column")
|
||||
private Integer referenceColumn;
|
||||
|
||||
@Column(name = "counterparty_column")
|
||||
private Integer counterpartyColumn;
|
||||
|
||||
@Column(name = "iban_column")
|
||||
private Integer ibanColumn;
|
||||
|
||||
@Column(name = "delimiter", nullable = false, length = 5)
|
||||
private String delimiter = ";";
|
||||
|
||||
/** Pattern compatible with {@code DateTimeFormatter.ofPattern}, e.g. {@code dd.MM.yyyy}. */
|
||||
@Column(name = "date_format", nullable = false, length = 20)
|
||||
private String dateFormat = "dd.MM.yyyy";
|
||||
|
||||
/** Single character — typically "," (German) or "." (English). */
|
||||
@Column(name = "decimal_separator", nullable = false, length = 1)
|
||||
private String decimalSeparator = ",";
|
||||
|
||||
@Column(name = "skip_header_rows", nullable = false)
|
||||
private Integer skipHeaderRows = 1;
|
||||
|
||||
/** Character set name, e.g. {@code ISO-8859-1} (German default), {@code UTF-8}, {@code windows-1252}. */
|
||||
@Column(name = "encoding", nullable = false, length = 20)
|
||||
private String encoding = "ISO-8859-1";
|
||||
|
||||
@Column(name = "is_default", nullable = false)
|
||||
private Boolean isDefault = false;
|
||||
|
||||
// Getters and setters
|
||||
|
||||
public UUID getClubId() { return clubId; }
|
||||
public void setClubId(UUID clubId) { this.clubId = clubId; }
|
||||
|
||||
public String getName() { return name; }
|
||||
public void setName(String name) { this.name = name; }
|
||||
|
||||
public Integer getDateColumn() { return dateColumn; }
|
||||
public void setDateColumn(Integer dateColumn) { this.dateColumn = dateColumn; }
|
||||
|
||||
public Integer getAmountColumn() { return amountColumn; }
|
||||
public void setAmountColumn(Integer amountColumn) { this.amountColumn = amountColumn; }
|
||||
|
||||
public Integer getReferenceColumn() { return referenceColumn; }
|
||||
public void setReferenceColumn(Integer referenceColumn) { this.referenceColumn = referenceColumn; }
|
||||
|
||||
public Integer getCounterpartyColumn() { return counterpartyColumn; }
|
||||
public void setCounterpartyColumn(Integer counterpartyColumn) { this.counterpartyColumn = counterpartyColumn; }
|
||||
|
||||
public Integer getIbanColumn() { return ibanColumn; }
|
||||
public void setIbanColumn(Integer ibanColumn) { this.ibanColumn = ibanColumn; }
|
||||
|
||||
public String getDelimiter() { return delimiter; }
|
||||
public void setDelimiter(String delimiter) { this.delimiter = delimiter; }
|
||||
|
||||
public String getDateFormat() { return dateFormat; }
|
||||
public void setDateFormat(String dateFormat) { this.dateFormat = dateFormat; }
|
||||
|
||||
public String getDecimalSeparator() { return decimalSeparator; }
|
||||
public void setDecimalSeparator(String decimalSeparator) { this.decimalSeparator = decimalSeparator; }
|
||||
|
||||
public Integer getSkipHeaderRows() { return skipHeaderRows; }
|
||||
public void setSkipHeaderRows(Integer skipHeaderRows) { this.skipHeaderRows = skipHeaderRows; }
|
||||
|
||||
public String getEncoding() { return encoding; }
|
||||
public void setEncoding(String encoding) { this.encoding = encoding; }
|
||||
|
||||
public Boolean getIsDefault() { return isDefault; }
|
||||
public void setIsDefault(Boolean isDefault) { this.isDefault = isDefault; }
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package de.cannamanage.domain.entity;
|
||||
import de.cannamanage.domain.enums.MemberStatus;
|
||||
import jakarta.persistence.*;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.time.LocalDate;
|
||||
import java.util.UUID;
|
||||
|
||||
@@ -49,6 +50,17 @@ public class Member extends AbstractTenantEntity {
|
||||
@Column(name = "prevention_officer", nullable = false)
|
||||
private boolean preventionOfficer = false;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Member's IBAN, used by the bank statement matching engine.
|
||||
* Nullable: only populated after the member explicitly grants BANK_DATA consent.
|
||||
*/
|
||||
@Column(name = "iban", length = 34)
|
||||
private String iban;
|
||||
|
||||
/** Sprint 10 — Timestamp when BANK_DATA consent was granted for this IBAN. */
|
||||
@Column(name = "iban_consent_date")
|
||||
private Instant ibanConsentDate;
|
||||
|
||||
public UUID getUserId() { return userId; }
|
||||
public void setUserId(UUID userId) { this.userId = userId; }
|
||||
|
||||
@@ -81,4 +93,10 @@ public class Member extends AbstractTenantEntity {
|
||||
|
||||
public boolean isPreventionOfficer() { return preventionOfficer; }
|
||||
public void setPreventionOfficer(boolean preventionOfficer) { this.preventionOfficer = preventionOfficer; }
|
||||
|
||||
public String getIban() { return iban; }
|
||||
public void setIban(String iban) { this.iban = iban; }
|
||||
|
||||
public Instant getIbanConsentDate() { return ibanConsentDate; }
|
||||
public void setIbanConsentDate(Instant ibanConsentDate) { this.ibanConsentDate = ibanConsentDate; }
|
||||
}
|
||||
|
||||
@@ -93,5 +93,11 @@ public enum AuditEventType {
|
||||
AUTHORITY_EXPORT,
|
||||
DESTRUCTION_RECORDED,
|
||||
TRANSPORT_RECORDED,
|
||||
RETENTION_DELETED
|
||||
RETENTION_DELETED,
|
||||
|
||||
// Sprint 10 — Smart Payment Import events
|
||||
BANK_IMPORT_STARTED,
|
||||
BANK_IMPORT_COMPLETED,
|
||||
BANK_IMPORT_FAILED,
|
||||
BANK_PAYMENT_CONFIRMED
|
||||
}
|
||||
|
||||
@@ -0,0 +1,16 @@
|
||||
package de.cannamanage.domain.enums;
|
||||
|
||||
/**
|
||||
* Supported bank statement file formats for the payment import feature (Sprint 10).
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@link #MT940} — SWIFT MT940, the legacy German banking standard (CSV-like text format)</li>
|
||||
* <li>{@link #CAMT053} — ISO 20022 camt.053 Bank-to-Customer Statement (XML)</li>
|
||||
* <li>{@link #CSV} — Generic delimited text, requires a {@code CsvColumnMapping} to interpret</li>
|
||||
* </ul>
|
||||
*/
|
||||
public enum BankFormat {
|
||||
MT940,
|
||||
CAMT053,
|
||||
CSV
|
||||
}
|
||||
@@ -9,5 +9,7 @@ public enum ConsentType {
|
||||
ANALYTICS,
|
||||
// Sprint 7 — Push notification consent (GDPR Art. 7(1))
|
||||
NOTIFICATION_PUSH,
|
||||
NOTIFICATION_EMAIL
|
||||
NOTIFICATION_EMAIL,
|
||||
// Sprint 10 — IBAN storage consent (DSGVO Art. 6(1)(a)) for bank statement matching
|
||||
BANK_DATA
|
||||
}
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
package de.cannamanage.domain.enums;
|
||||
|
||||
/**
|
||||
* Lifecycle status of a {@code BankImportSession} (Sprint 10).
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@link #PENDING} — file uploaded + parsed, matching not yet run</li>
|
||||
* <li>{@link #IN_REVIEW} — matching completed, admin is reviewing/confirming entries</li>
|
||||
* <li>{@link #COMPLETED} — admin finalized the session; session is immutable per GoBD</li>
|
||||
* <li>{@link #FAILED} — parse or processing error, or admin discarded an in-progress session</li>
|
||||
* </ul>
|
||||
*/
|
||||
public enum ImportSessionStatus {
|
||||
PENDING,
|
||||
IN_REVIEW,
|
||||
COMPLETED,
|
||||
FAILED
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package de.cannamanage.domain.enums;
|
||||
|
||||
/**
|
||||
* Match status of a {@code BankTransaction} against the membership/payment registry (Sprint 10).
|
||||
*
|
||||
* <ul>
|
||||
* <li>{@link #UNMATCHED} — no member candidate found by the matching engine</li>
|
||||
* <li>{@link #SUGGESTED} — a candidate was found with medium confidence (60-89%) — admin review required</li>
|
||||
* <li>{@link #MATCHED} — a candidate was auto-matched with high confidence (≥90%) — pre-selected for confirm</li>
|
||||
* <li>{@link #CONFIRMED} — admin confirmed the match; a {@code Payment} has been created</li>
|
||||
* <li>{@link #SKIPPED} — admin explicitly skipped this transaction (not a member payment)</li>
|
||||
* </ul>
|
||||
*/
|
||||
public enum MatchStatus {
|
||||
UNMATCHED,
|
||||
SUGGESTED,
|
||||
MATCHED,
|
||||
CONFIRMED,
|
||||
SKIPPED
|
||||
}
|
||||
@@ -5,5 +5,12 @@ public enum MemberStatus {
|
||||
SUSPENDED,
|
||||
EXPELLED,
|
||||
PENDING_APPROVAL,
|
||||
RESIGNED
|
||||
RESIGNED,
|
||||
/**
|
||||
* Sprint 9 retention semantics — used by {@code RetentionService} to identify
|
||||
* members who have left the club and whose data falls under the KCanG §24
|
||||
* 5-year retention rule. Treated equivalently to {@link #RESIGNED} for
|
||||
* membership-lifecycle purposes.
|
||||
*/
|
||||
LEFT
|
||||
}
|
||||
|
||||
@@ -28,5 +28,7 @@ public enum NotificationType {
|
||||
BOARD_TERM_EXPIRING,
|
||||
// Sprint 9 — Compliance:
|
||||
COMPLIANCE_DEADLINE,
|
||||
RETENTION_WARNING
|
||||
RETENTION_WARNING,
|
||||
// Sprint 10 — Smart Payment Import:
|
||||
BANK_IMPORT_COMPLETED
|
||||
}
|
||||
|
||||
@@ -26,5 +26,8 @@ public enum StaffPermission {
|
||||
// Sprint 9:
|
||||
GENERATE_REPORTS,
|
||||
VIEW_COMPLIANCE,
|
||||
MANAGE_COMPLIANCE
|
||||
MANAGE_COMPLIANCE,
|
||||
// Sprint 10 — Smart Payment Import:
|
||||
/** Permission to upload bank statements and confirm matched transactions. */
|
||||
FINANCE_IMPORT
|
||||
}
|
||||
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
package de.cannamanage.service.bankimport;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Unrecoverable error while parsing a bank statement file.
|
||||
* <p>
|
||||
* Thrown by {@link BankStatementParser#parse} when the input cannot be processed
|
||||
* (malformed XML, unreadable encoding, missing required fields, etc.).
|
||||
* <p>
|
||||
* Recoverable issues (skipped lines, missing optional fields) are reported as
|
||||
* warnings on the {@link ParseResult} instead of throwing.
|
||||
*/
|
||||
public class BankStatementParseException extends RuntimeException {
|
||||
|
||||
public BankStatementParseException(String message) {
|
||||
super(message);
|
||||
}
|
||||
|
||||
public BankStatementParseException(String message, Throwable cause) {
|
||||
super(message, cause);
|
||||
}
|
||||
}
|
||||
+59
@@ -0,0 +1,59 @@
|
||||
package de.cannamanage.service.bankimport;
|
||||
|
||||
import de.cannamanage.domain.entity.CsvColumnMapping;
|
||||
import de.cannamanage.domain.enums.BankFormat;
|
||||
|
||||
import java.io.InputStream;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Strategy interface for parsing one bank statement format.
|
||||
* <p>
|
||||
* Implementations are Spring {@code @Component} beans, auto-discovered by
|
||||
* {@link BankStatementParserService}. Add a new format by adding a new
|
||||
* implementation — no further wiring needed.
|
||||
* <p>
|
||||
* Each parser must:
|
||||
* <ul>
|
||||
* <li>Declare the single format it handles via {@link #getSupportedFormat()}</li>
|
||||
* <li>Detect that format reliably from the first ~512 bytes of file content via
|
||||
* {@link #canParse(String, byte[])}</li>
|
||||
* <li>Stream-parse the full file via {@link #parse(InputStream, String, CsvColumnMapping)},
|
||||
* producing a {@link ParseResult} — must not load the entire file into memory
|
||||
* (large bank exports can exceed 50 MB)</li>
|
||||
* </ul>
|
||||
*/
|
||||
public interface BankStatementParser {
|
||||
|
||||
/** The single bank statement format this parser handles. */
|
||||
BankFormat getSupportedFormat();
|
||||
|
||||
/**
|
||||
* Probe whether this parser can handle the given file.
|
||||
* <p>
|
||||
* Implementations should inspect the filename extension and/or the first bytes
|
||||
* (typically the first 512). Must be fast and side-effect free — called for
|
||||
* every uploaded file by {@link BankStatementParserService#detectFormat}.
|
||||
*
|
||||
* @param filename original upload filename (after path sanitization)
|
||||
* @param headerBytes first bytes of the file content (at least 512 bytes if available)
|
||||
* @return {@code true} if this parser claims the format, {@code false} otherwise
|
||||
*/
|
||||
boolean canParse(String filename, byte[] headerBytes);
|
||||
|
||||
/**
|
||||
* Parse the full statement.
|
||||
* <p>
|
||||
* The {@code mapping} parameter is required for {@link BankFormat#CSV} and
|
||||
* ignored by structured formats (MT940, CAMT.053). Callers must pass a non-null
|
||||
* mapping when the detected format is CSV.
|
||||
* <p>
|
||||
* The input stream is consumed but not closed — callers own the stream lifecycle.
|
||||
*
|
||||
* @param inputStream stream over the file content (caller closes)
|
||||
* @param filename original filename, used only for warning/error messages
|
||||
* @param mapping CSV column mapping (required for CSV, may be {@code null} otherwise)
|
||||
* @return parsed result; never {@code null}
|
||||
* @throws BankStatementParseException on unrecoverable parse errors
|
||||
*/
|
||||
ParseResult parse(InputStream inputStream, String filename, CsvColumnMapping mapping);
|
||||
}
|
||||
+234
@@ -0,0 +1,234 @@
|
||||
package de.cannamanage.service.bankimport;
|
||||
|
||||
import de.cannamanage.domain.entity.CsvColumnMapping;
|
||||
import de.cannamanage.domain.enums.BankFormat;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.SequenceInputStream;
|
||||
import java.util.EnumMap;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.Objects;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Façade that detects a bank statement's format and routes parsing
|
||||
* to the matching {@link BankStatementParser}.
|
||||
* <p>
|
||||
* Spring auto-discovers all {@link BankStatementParser} beans and injects them
|
||||
* here as a {@code List}, so adding a new format only requires adding a new
|
||||
* parser bean. The service indexes them once at construction time keyed by
|
||||
* {@link BankFormat} for O(1) dispatch.
|
||||
* <p>
|
||||
* Format detection is a two-step probe:
|
||||
* <ol>
|
||||
* <li>Filename-extension hint ({@code .xml} → CAMT, {@code .sta}/{@code .mt940}/{@code .swift}
|
||||
* → MT940, {@code .csv}/{@code .txt} → CSV). The hint biases probe order
|
||||
* but never overrides content.</li>
|
||||
* <li>Content probe via {@link BankStatementParser#canParse(String, byte[])}
|
||||
* against the first {@value #HEADER_PROBE_BYTES} bytes. The first parser
|
||||
* that claims the content wins.</li>
|
||||
* </ol>
|
||||
* If no parser claims the file an {@link UnrecognizedFormatException} is thrown.
|
||||
*/
|
||||
@Service
|
||||
public class BankStatementParserService {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(BankStatementParserService.class);
|
||||
|
||||
/** Bytes inspected for content-based format detection. */
|
||||
public static final int HEADER_PROBE_BYTES = 512;
|
||||
|
||||
private final Map<BankFormat, BankStatementParser> parsersByFormat;
|
||||
private final List<BankStatementParser> parsers;
|
||||
|
||||
public BankStatementParserService(List<BankStatementParser> parsers) {
|
||||
Objects.requireNonNull(parsers, "parsers");
|
||||
this.parsers = List.copyOf(parsers);
|
||||
Map<BankFormat, BankStatementParser> index = new EnumMap<>(BankFormat.class);
|
||||
for (BankStatementParser parser : parsers) {
|
||||
BankFormat format = parser.getSupportedFormat();
|
||||
BankStatementParser existing = index.put(format, parser);
|
||||
if (existing != null) {
|
||||
throw new IllegalStateException(
|
||||
"Duplicate BankStatementParser beans for format " + format
|
||||
+ ": " + existing.getClass().getName()
|
||||
+ " and " + parser.getClass().getName());
|
||||
}
|
||||
}
|
||||
this.parsersByFormat = Map.copyOf(index);
|
||||
log.info("BankStatementParserService initialised with {} parsers: {}",
|
||||
parsersByFormat.size(), parsersByFormat.keySet());
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect the {@link BankFormat} of a file based on filename and a content probe.
|
||||
* <p>
|
||||
* The {@code content} array may be the full file content or just a head slice —
|
||||
* only the first {@value #HEADER_PROBE_BYTES} bytes are inspected.
|
||||
*
|
||||
* @param filename original upload filename (used for extension hint and error messages)
|
||||
* @param content file content (or at least a head slice); must not be {@code null}
|
||||
* @return the detected format
|
||||
* @throws UnrecognizedFormatException if no registered parser claims the file
|
||||
*/
|
||||
public BankFormat detectFormat(String filename, byte[] content) {
|
||||
Objects.requireNonNull(content, "content");
|
||||
String safeName = filename == null ? "" : filename;
|
||||
byte[] header = sliceHeader(content);
|
||||
|
||||
// 1. Extension hint — try the hinted parser first, but never trust extension alone.
|
||||
BankFormat hint = extensionHint(safeName);
|
||||
if (hint != null) {
|
||||
BankStatementParser hinted = parsersByFormat.get(hint);
|
||||
if (hinted != null && hinted.canParse(safeName, header)) {
|
||||
log.debug("Format detected via extension hint: {} → {}", safeName, hint);
|
||||
return hint;
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Content probe — ask every parser, hinted one already tried.
|
||||
for (BankStatementParser parser : parsers) {
|
||||
BankFormat format = parser.getSupportedFormat();
|
||||
if (format == hint) {
|
||||
continue; // already tried
|
||||
}
|
||||
if (parser.canParse(safeName, header)) {
|
||||
log.debug("Format detected via content probe: {} → {}", safeName, format);
|
||||
return format;
|
||||
}
|
||||
}
|
||||
|
||||
throw new UnrecognizedFormatException(
|
||||
"Bank statement format could not be detected for file: " + safeName);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a bank statement using the parser registered for the given format.
|
||||
* <p>
|
||||
* The {@code csvMapping} argument is required when {@code format} is
|
||||
* {@link BankFormat#CSV} and ignored otherwise. The input stream is consumed
|
||||
* but not closed — the caller owns the stream lifecycle.
|
||||
*
|
||||
* @param input file content stream (caller closes)
|
||||
* @param filename original filename for warnings and error messages
|
||||
* @param format pre-detected format (typically the result of {@link #detectFormat})
|
||||
* @param csvMapping CSV column mapping, required iff {@code format == CSV}
|
||||
* @return parse result; never {@code null}
|
||||
* @throws UnrecognizedFormatException if no parser is registered for the format
|
||||
* @throws BankStatementParseException on parser-level errors
|
||||
* @throws IllegalArgumentException if {@code csvMapping} is {@code null} for CSV
|
||||
*/
|
||||
public ParseResult parse(InputStream input, String filename, BankFormat format, CsvColumnMapping csvMapping) {
|
||||
Objects.requireNonNull(input, "input");
|
||||
Objects.requireNonNull(format, "format");
|
||||
if (format == BankFormat.CSV && csvMapping == null) {
|
||||
throw new IllegalArgumentException("csvMapping is required for CSV format");
|
||||
}
|
||||
BankStatementParser parser = parsersByFormat.get(format);
|
||||
if (parser == null) {
|
||||
throw new UnrecognizedFormatException(
|
||||
"No parser registered for format " + format
|
||||
+ " (available: " + parsersByFormat.keySet() + ")");
|
||||
}
|
||||
log.debug("Parsing {} with {} ({} bytes mapping={})",
|
||||
filename, parser.getClass().getSimpleName(), format,
|
||||
csvMapping == null ? "n/a" : csvMapping.getId());
|
||||
return parser.parse(input, filename, csvMapping);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convenience: detect the format from the file content and parse it in one call.
|
||||
* <p>
|
||||
* The content is buffered in memory (callers must size accordingly) and replayed
|
||||
* through the matching parser. For very large files prefer the two-step API:
|
||||
* read a small header into memory, call {@link #detectFormat}, then call
|
||||
* {@link #parse(InputStream, String, BankFormat, CsvColumnMapping)} with the
|
||||
* full streaming source.
|
||||
*
|
||||
* @param content full file content
|
||||
* @param filename original filename
|
||||
* @param csvMapping CSV mapping (required iff CSV is detected)
|
||||
* @return parse result
|
||||
*/
|
||||
public ParseResult detectAndParse(byte[] content, String filename, CsvColumnMapping csvMapping) {
|
||||
Objects.requireNonNull(content, "content");
|
||||
BankFormat format = detectFormat(filename, content);
|
||||
try (InputStream in = new ByteArrayInputStream(content)) {
|
||||
return parse(in, filename, format, csvMapping);
|
||||
} catch (IOException e) {
|
||||
// ByteArrayInputStream#close is a no-op, but the contract still declares IOException.
|
||||
throw new BankStatementParseException("Failed to close in-memory stream for " + filename, e);
|
||||
}
|
||||
}
|
||||
|
||||
/** Returns the immutable set of formats this service can dispatch to. */
|
||||
public java.util.Set<BankFormat> supportedFormats() {
|
||||
return parsersByFormat.keySet();
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------------
|
||||
// helpers
|
||||
// ------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Returns at most {@value #HEADER_PROBE_BYTES} bytes from the start of {@code content}.
|
||||
* Never copies more than necessary.
|
||||
*/
|
||||
private static byte[] sliceHeader(byte[] content) {
|
||||
if (content.length <= HEADER_PROBE_BYTES) {
|
||||
return content;
|
||||
}
|
||||
byte[] head = new byte[HEADER_PROBE_BYTES];
|
||||
System.arraycopy(content, 0, head, 0, HEADER_PROBE_BYTES);
|
||||
return head;
|
||||
}
|
||||
|
||||
/**
|
||||
* Map a filename extension to a likely {@link BankFormat}.
|
||||
* Returns {@code null} when no hint can be derived.
|
||||
*/
|
||||
private static BankFormat extensionHint(String filename) {
|
||||
int dot = filename.lastIndexOf('.');
|
||||
if (dot < 0 || dot == filename.length() - 1) {
|
||||
return null;
|
||||
}
|
||||
String ext = filename.substring(dot + 1).toLowerCase(Locale.ROOT);
|
||||
return switch (ext) {
|
||||
case "xml", "camt", "053" -> BankFormat.CAMT053;
|
||||
case "mt940", "sta", "swift" -> BankFormat.MT940;
|
||||
case "csv", "txt", "tsv" -> BankFormat.CSV;
|
||||
default -> null;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Wraps two streams so a previously-buffered header can be replayed in front
|
||||
* of a still-open file stream — handy when callers want to probe + parse a
|
||||
* single source without re-reading the disk. Currently unused by the public
|
||||
* API; kept as a building block for future controllers/services.
|
||||
*/
|
||||
@SuppressWarnings("unused")
|
||||
static InputStream concat(byte[] head, InputStream rest) {
|
||||
return new SequenceInputStream(new ByteArrayInputStream(head), rest);
|
||||
}
|
||||
|
||||
/**
|
||||
* Thrown when no registered {@link BankStatementParser} claims a file.
|
||||
* Distinct from {@link BankStatementParseException} so callers can
|
||||
* surface "Format nicht erkannt" as a 400-style validation error rather
|
||||
* than a 500-style parse failure.
|
||||
*/
|
||||
public static class UnrecognizedFormatException extends RuntimeException {
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
public UnrecognizedFormatException(String message) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
}
|
||||
+342
@@ -0,0 +1,342 @@
|
||||
package de.cannamanage.service.bankimport;
|
||||
|
||||
import de.cannamanage.domain.entity.CsvColumnMapping;
|
||||
import de.cannamanage.domain.enums.BankFormat;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import javax.xml.stream.XMLInputFactory;
|
||||
import javax.xml.stream.XMLStreamConstants;
|
||||
import javax.xml.stream.XMLStreamException;
|
||||
import javax.xml.stream.XMLStreamReader;
|
||||
import java.io.InputStream;
|
||||
import java.math.BigDecimal;
|
||||
import java.math.RoundingMode;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.LocalDate;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import java.util.ArrayList;
|
||||
import java.util.ArrayDeque;
|
||||
import java.util.Deque;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Parser for ISO 20022 CAMT.053 bank-to-customer statements.
|
||||
* <p>
|
||||
* CAMT.053 is the modern XML successor to MT940, mandated for SEPA transfers
|
||||
* and used by all German banks for online-banking exports since ~2014. It
|
||||
* exists in multiple schema versions (camt.053.001.02 through camt.053.001.10);
|
||||
* we handle them generically by ignoring the namespace and matching local names —
|
||||
* the elements we read have been stable across versions.
|
||||
* <p>
|
||||
* Document structure (simplified):
|
||||
* <pre>
|
||||
* Document
|
||||
* BkToCstmrStmt
|
||||
* Stmt ← one or more statements
|
||||
* Id, ElctrncSeqNb, CreDtTm, FrToDt
|
||||
* Acct/Id/IBAN ← account IBAN
|
||||
* Bal ← opening/closing balances
|
||||
* Ntry ← one entry per transaction
|
||||
* Amt ← amount with @Ccy attribute
|
||||
* CdtDbtInd ← CRDT or DBIT
|
||||
* BookgDt/Dt, ValDt/Dt
|
||||
* NtryRef ← bank reference
|
||||
* NtryDtls/TxDtls
|
||||
* RmtInf/Ustrd ← unstructured remittance ("Verwendungszweck")
|
||||
* RltdPties/Dbtr/Nm, /Cdtr/Nm
|
||||
* RltdPties/DbtrAcct/Id/IBAN, /CdtrAcct/Id/IBAN
|
||||
* </pre>
|
||||
* <p>
|
||||
* <strong>XXE hardening (Security advisory):</strong> The {@link XMLInputFactory} is
|
||||
* configured to disable DTDs, external entities, and entity reference expansion.
|
||||
* This makes the parser safe against XXE attacks (CWE-611) — bank statements arrive
|
||||
* from arbitrary external sources, so this is non-negotiable.
|
||||
* <p>
|
||||
* <strong>Streaming:</strong> StAX is used (not DOM/JAXB) to keep memory bounded —
|
||||
* large CAMT files from active accounts can exceed 50 MB with thousands of entries.
|
||||
*/
|
||||
@Component
|
||||
public class Camt053Parser implements BankStatementParser {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(Camt053Parser.class);
|
||||
|
||||
private static final DateTimeFormatter ISO_DATE = DateTimeFormatter.ISO_LOCAL_DATE;
|
||||
|
||||
private final XMLInputFactory xmlInputFactory;
|
||||
|
||||
public Camt053Parser() {
|
||||
this.xmlInputFactory = XMLInputFactory.newFactory();
|
||||
// Disable DTD support entirely — there is no legitimate DTD in CAMT.053.
|
||||
// This is the primary XXE defence: no DOCTYPE, no entities, no external resources.
|
||||
this.xmlInputFactory.setProperty(XMLInputFactory.SUPPORT_DTD, false);
|
||||
this.xmlInputFactory.setProperty(XMLInputFactory.IS_SUPPORTING_EXTERNAL_ENTITIES, false);
|
||||
// Even with DTDs disabled, set these as belt-and-braces — some parser impls
|
||||
// honour them independently.
|
||||
try {
|
||||
this.xmlInputFactory.setProperty(XMLInputFactory.IS_REPLACING_ENTITY_REFERENCES, false);
|
||||
} catch (IllegalArgumentException ignored) {
|
||||
// Not all StAX implementations expose this property.
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public BankFormat getSupportedFormat() {
|
||||
return BankFormat.CAMT053;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean canParse(String filename, byte[] headerBytes) {
|
||||
if (headerBytes == null || headerBytes.length == 0) {
|
||||
return false;
|
||||
}
|
||||
String head = new String(headerBytes, StandardCharsets.UTF_8);
|
||||
// Must look like XML and contain either the camt.053 namespace marker
|
||||
// or the unmistakable BkToCstmrStmt root element. Either alone is sufficient.
|
||||
boolean looksLikeXml = head.contains("<?xml") || head.trim().startsWith("<");
|
||||
if (!looksLikeXml) return false;
|
||||
return head.contains("camt.053") || head.contains("BkToCstmrStmt");
|
||||
}
|
||||
|
||||
@Override
|
||||
public ParseResult parse(InputStream inputStream, String filename, CsvColumnMapping mapping) {
|
||||
List<ParsedTransaction> transactions = new ArrayList<>();
|
||||
List<String> warnings = new ArrayList<>();
|
||||
|
||||
String accountIban = null;
|
||||
LocalDate statementDate = null;
|
||||
Integer openingBalanceCents = null;
|
||||
Integer closingBalanceCents = null;
|
||||
|
||||
// Path stack — used to disambiguate elements with the same local name in
|
||||
// different contexts (e.g. <Amt> on the entry vs. inside <TxDtls/InstdAmt>).
|
||||
Deque<String> path = new ArrayDeque<>();
|
||||
|
||||
XMLStreamReader reader = null;
|
||||
try {
|
||||
reader = xmlInputFactory.createXMLStreamReader(inputStream, "UTF-8");
|
||||
|
||||
// In-progress entry state. We commit on </Ntry>.
|
||||
EntryAccumulator current = null;
|
||||
// In-progress balance state. We commit on </Bal>.
|
||||
BalanceAccumulator currentBal = null;
|
||||
// Buffer for the most recent character data (only meaningful inside leaf elements).
|
||||
StringBuilder chars = new StringBuilder();
|
||||
String currentAmtCurrency = "EUR";
|
||||
|
||||
while (reader.hasNext()) {
|
||||
int event = reader.next();
|
||||
switch (event) {
|
||||
case XMLStreamConstants.START_ELEMENT -> {
|
||||
String local = reader.getLocalName();
|
||||
path.push(local);
|
||||
chars.setLength(0);
|
||||
|
||||
if ("Ntry".equals(local)) {
|
||||
current = new EntryAccumulator();
|
||||
} else if ("Bal".equals(local)) {
|
||||
currentBal = new BalanceAccumulator();
|
||||
} else if ("Amt".equals(local) && (insideBal(path) || directChildOf(path, "Ntry"))) {
|
||||
// Capture the @Ccy attribute on this Amt element only.
|
||||
currentAmtCurrency = reader.getAttributeValue(null, "Ccy");
|
||||
if (currentAmtCurrency == null) currentAmtCurrency = "EUR";
|
||||
}
|
||||
}
|
||||
case XMLStreamConstants.CHARACTERS, XMLStreamConstants.CDATA -> {
|
||||
chars.append(reader.getText());
|
||||
}
|
||||
case XMLStreamConstants.END_ELEMENT -> {
|
||||
String local = reader.getLocalName();
|
||||
String text = chars.toString().strip();
|
||||
|
||||
// --- Account IBAN (statement-level) ---
|
||||
if ("IBAN".equals(local) && pathContains(path, "Acct") && !pathContains(path, "RltdPties")) {
|
||||
if (accountIban == null) accountIban = text;
|
||||
}
|
||||
// --- Statement period (use the ToDtTm/Dt as statementDate fallback) ---
|
||||
else if (("ToDt".equals(local) || "Dt".equals(local))
|
||||
&& pathContains(path, "FrToDt") && statementDate == null) {
|
||||
statementDate = tryParseDate(text);
|
||||
}
|
||||
|
||||
// --- Balances ---
|
||||
if (currentBal != null) {
|
||||
if ("Cd".equals(local) && pathContains(path, "Tp") && pathContains(path, "Bal")) {
|
||||
currentBal.code = text; // OPBD/CLBD/PRCD/etc.
|
||||
} else if ("Amt".equals(local) && directChildOf(path, "Bal")) {
|
||||
currentBal.amountCents = parseAmountToCents(text);
|
||||
} else if ("CdtDbtInd".equals(local) && directChildOf(path, "Bal")) {
|
||||
currentBal.creditIndicator = "CRDT".equals(text);
|
||||
} else if ("Dt".equals(local) && pathContains(path, "Bal")) {
|
||||
currentBal.date = tryParseDate(text);
|
||||
} else if ("Bal".equals(local)) {
|
||||
// Commit
|
||||
Integer signedCents = currentBal.amountCents == null ? null
|
||||
: (currentBal.creditIndicator ? currentBal.amountCents : -currentBal.amountCents);
|
||||
if (signedCents != null && currentBal.code != null) {
|
||||
switch (currentBal.code) {
|
||||
case "OPBD", "PRCD" -> {
|
||||
if (openingBalanceCents == null) openingBalanceCents = signedCents;
|
||||
}
|
||||
case "CLBD" -> closingBalanceCents = signedCents;
|
||||
default -> { /* ITBD/CLAV/etc. — ignore */ }
|
||||
}
|
||||
if (statementDate == null && "CLBD".equals(currentBal.code)) {
|
||||
statementDate = currentBal.date;
|
||||
}
|
||||
}
|
||||
currentBal = null;
|
||||
}
|
||||
}
|
||||
|
||||
// --- Entry fields ---
|
||||
if (current != null) {
|
||||
if ("Amt".equals(local) && directChildOf(path, "Ntry")) {
|
||||
current.amountCents = parseAmountToCents(text);
|
||||
current.currency = currentAmtCurrency;
|
||||
} else if ("CdtDbtInd".equals(local) && directChildOf(path, "Ntry")) {
|
||||
current.isCredit = "CRDT".equals(text);
|
||||
} else if ("Dt".equals(local) && pathContains(path, "BookgDt")) {
|
||||
current.bookingDate = tryParseDate(text);
|
||||
} else if ("Dt".equals(local) && pathContains(path, "ValDt")) {
|
||||
current.valueDate = tryParseDate(text);
|
||||
} else if ("NtryRef".equals(local) && directChildOf(path, "Ntry")) {
|
||||
current.bankReference = text;
|
||||
} else if ("AcctSvcrRef".equals(local) && directChildOf(path, "Ntry")
|
||||
&& current.bankReference == null) {
|
||||
// Fallback when NtryRef is absent.
|
||||
current.bankReference = text;
|
||||
} else if ("Ustrd".equals(local) && pathContains(path, "RmtInf")) {
|
||||
// Multiple <Ustrd> elements are concatenated per ISO 20022.
|
||||
if (current.referenceText == null) {
|
||||
current.referenceText = text;
|
||||
} else {
|
||||
current.referenceText = current.referenceText + " " + text;
|
||||
}
|
||||
} else if ("Nm".equals(local)) {
|
||||
// Counterparty name: <Dbtr><Nm> for credits, <Cdtr><Nm> for debits.
|
||||
// We grab the name from the side that's NOT us.
|
||||
if (current.isCredit != null && current.isCredit && pathContains(path, "Dbtr")
|
||||
&& !pathContains(path, "DbtrAgt")) {
|
||||
current.counterpartyName = text;
|
||||
} else if (current.isCredit != null && !current.isCredit && pathContains(path, "Cdtr")
|
||||
&& !pathContains(path, "CdtrAgt")) {
|
||||
current.counterpartyName = text;
|
||||
}
|
||||
} else if ("IBAN".equals(local) && pathContains(path, "RltdPties")) {
|
||||
if (current.isCredit != null && current.isCredit && pathContains(path, "DbtrAcct")) {
|
||||
current.counterpartyIban = text;
|
||||
} else if (current.isCredit != null && !current.isCredit && pathContains(path, "CdtrAcct")) {
|
||||
current.counterpartyIban = text;
|
||||
}
|
||||
} else if ("Ntry".equals(local)) {
|
||||
// Commit entry.
|
||||
if (current.bookingDate != null && current.amountCents != null
|
||||
&& current.isCredit != null) {
|
||||
int signed = current.isCredit ? current.amountCents : -current.amountCents;
|
||||
LocalDate value = current.valueDate != null ? current.valueDate : current.bookingDate;
|
||||
transactions.add(new ParsedTransaction(
|
||||
current.bookingDate, value, signed,
|
||||
current.currency != null ? current.currency : "EUR",
|
||||
current.referenceText, current.counterpartyName,
|
||||
current.counterpartyIban, current.bankReference));
|
||||
} else {
|
||||
warnings.add("CAMT.053: skipped Ntry with missing required fields "
|
||||
+ "(bookingDate=" + current.bookingDate + ", amount=" + current.amountCents
|
||||
+ ", credit=" + current.isCredit + ")");
|
||||
}
|
||||
current = null;
|
||||
}
|
||||
}
|
||||
|
||||
// Pop the path AFTER all the directChildOf/pathContains checks above
|
||||
// have used the still-pushed element as their context.
|
||||
path.pop();
|
||||
chars.setLength(0);
|
||||
}
|
||||
default -> { /* skip comments, PIs, whitespace events between elements */ }
|
||||
}
|
||||
}
|
||||
} catch (XMLStreamException e) {
|
||||
throw new BankStatementParseException("CAMT.053 XML error in " + filename + ": " + e.getMessage(), e);
|
||||
} finally {
|
||||
if (reader != null) {
|
||||
try { reader.close(); } catch (XMLStreamException ignored) { }
|
||||
}
|
||||
}
|
||||
|
||||
log.debug("CAMT.053 parsed: file={} transactions={} warnings={}",
|
||||
filename, transactions.size(), warnings.size());
|
||||
return new ParseResult(transactions, accountIban, statementDate,
|
||||
openingBalanceCents, closingBalanceCents, warnings);
|
||||
}
|
||||
|
||||
/** Convert a decimal-dot amount string (ISO 20022 standard) to cents. */
|
||||
static int parseAmountToCents(String amount) {
|
||||
// ISO 20022 amounts use a period as decimal separator and never a thousands separator.
|
||||
// Examples: "50.00", "1234.56", "0.99", "100" (no fraction).
|
||||
BigDecimal value = new BigDecimal(amount.strip());
|
||||
return value.movePointRight(2).setScale(0, RoundingMode.HALF_UP).intValueExact();
|
||||
}
|
||||
|
||||
private static LocalDate tryParseDate(String text) {
|
||||
if (text == null || text.isEmpty()) return null;
|
||||
try {
|
||||
// CAMT uses xs:date (YYYY-MM-DD) for <Dt> and xs:dateTime for <DtTm>.
|
||||
// We accept either by stripping any time portion.
|
||||
int tIdx = text.indexOf('T');
|
||||
String dateOnly = tIdx > 0 ? text.substring(0, tIdx) : text;
|
||||
return LocalDate.parse(dateOnly, ISO_DATE);
|
||||
} catch (RuntimeException e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Is the current path inside a {@code <Bal>} subtree? */
|
||||
private static boolean insideBal(Deque<String> path) {
|
||||
return pathContains(path, "Bal");
|
||||
}
|
||||
|
||||
/**
|
||||
* Is the second-from-top of the path the given element? Used to disambiguate
|
||||
* {@code <Amt>} as a direct child of {@code <Ntry>} from {@code <Amt>} inside
|
||||
* a nested {@code <TxDtls>}.
|
||||
* <p>
|
||||
* Note: top of the path is the current element itself, so the parent is index 1.
|
||||
*/
|
||||
private static boolean directChildOf(Deque<String> path, String parent) {
|
||||
if (path.size() < 2) return false;
|
||||
var it = path.iterator();
|
||||
it.next(); // skip self
|
||||
return parent.equals(it.next());
|
||||
}
|
||||
|
||||
private static boolean pathContains(Deque<String> path, String element) {
|
||||
for (String p : path) {
|
||||
if (element.equals(p)) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Mutable holder accumulating one entry across many StAX events. */
|
||||
private static final class EntryAccumulator {
|
||||
LocalDate bookingDate;
|
||||
LocalDate valueDate;
|
||||
Integer amountCents;
|
||||
String currency;
|
||||
Boolean isCredit;
|
||||
String referenceText;
|
||||
String counterpartyName;
|
||||
String counterpartyIban;
|
||||
String bankReference;
|
||||
}
|
||||
|
||||
/** Mutable holder for an in-progress {@code <Bal>} subtree. */
|
||||
private static final class BalanceAccumulator {
|
||||
String code; // OPBD, CLBD, ITBD, PRCD, …
|
||||
Integer amountCents;
|
||||
boolean creditIndicator = true;
|
||||
LocalDate date;
|
||||
}
|
||||
}
|
||||
+273
@@ -0,0 +1,273 @@
|
||||
package de.cannamanage.service.bankimport;
|
||||
|
||||
import de.cannamanage.domain.entity.CsvColumnMapping;
|
||||
import de.cannamanage.domain.enums.BankFormat;
|
||||
import org.apache.commons.csv.CSVFormat;
|
||||
import org.apache.commons.csv.CSVParser;
|
||||
import org.apache.commons.csv.CSVRecord;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.io.Reader;
|
||||
import java.nio.charset.Charset;
|
||||
import java.nio.charset.IllegalCharsetNameException;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.nio.charset.UnsupportedCharsetException;
|
||||
import java.time.LocalDate;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Parser for generic CSV bank statement exports.
|
||||
* <p>
|
||||
* Unlike MT940 and CAMT.053, CSV has no standardized layout — every bank uses
|
||||
* its own column order, encoding, and number format. This parser is configured
|
||||
* per-import via a {@link CsvColumnMapping} entity which the admin sets up once
|
||||
* per bank (typical names: "Sparkasse Export", "DKB Online", "ING Umsätze").
|
||||
* <p>
|
||||
* Configurable aspects (all from {@link CsvColumnMapping}):
|
||||
* <ul>
|
||||
* <li><strong>Encoding</strong> — defaults to ISO-8859-1 (the German banking
|
||||
* standard); UTF-8 and Windows-1252 are also common</li>
|
||||
* <li><strong>Delimiter</strong> — typically {@code ;} for German exports,
|
||||
* {@code ,} for English-locale tools</li>
|
||||
* <li><strong>Skip header rows</strong> — banks often emit 1-5 metadata rows
|
||||
* before the actual transaction header</li>
|
||||
* <li><strong>Column indices</strong> — 0-based positions of date, amount,
|
||||
* reference, counterparty name, IBAN; reference/counterparty/IBAN are optional</li>
|
||||
* <li><strong>Date format</strong> — any pattern compatible with {@link DateTimeFormatter}</li>
|
||||
* <li><strong>Decimal separator</strong> — {@code ,} (German) or {@code .} (English).
|
||||
* The opposite character is treated as a thousands separator and stripped.</li>
|
||||
* </ul>
|
||||
* <p>
|
||||
* Sign convention: many CSV exports use a single signed amount column; some use
|
||||
* separate "Soll"/"Haben" columns. The current implementation supports only the
|
||||
* signed-amount style — a future enhancement can add a {@code creditDebitColumn}
|
||||
* field to {@link CsvColumnMapping} if a customer needs the two-column variant.
|
||||
*/
|
||||
@Component
|
||||
public class CsvBankParser implements BankStatementParser {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(CsvBankParser.class);
|
||||
|
||||
@Override
|
||||
public BankFormat getSupportedFormat() {
|
||||
return BankFormat.CSV;
|
||||
}
|
||||
|
||||
/**
|
||||
* Acts as the fallback detector — if it's not XML and not MT940, we try CSV.
|
||||
* The detection logic in {@link BankStatementParserService} runs CSV last so
|
||||
* we don't accidentally claim MT940 or CAMT files that happen to have a {@code .csv}
|
||||
* extension by mistake.
|
||||
*/
|
||||
@Override
|
||||
public boolean canParse(String filename, byte[] headerBytes) {
|
||||
if (filename == null || headerBytes == null) return false;
|
||||
String lower = filename.toLowerCase();
|
||||
// Primary signal: file extension. Bank CSV exports are reliably named.
|
||||
if (lower.endsWith(".csv") || lower.endsWith(".txt")) {
|
||||
return true;
|
||||
}
|
||||
// Secondary: content looks delimited and is NOT XML.
|
||||
String head = new String(headerBytes, 0, Math.min(headerBytes.length, 256),
|
||||
StandardCharsets.ISO_8859_1);
|
||||
if (head.contains("<?xml") || head.trim().startsWith("<")) return false;
|
||||
// Looks for at least one likely delimiter on the first line.
|
||||
int nl = head.indexOf('\n');
|
||||
String firstLine = nl >= 0 ? head.substring(0, nl) : head;
|
||||
return firstLine.contains(";") || firstLine.contains("\t") || firstLine.contains(",");
|
||||
}
|
||||
|
||||
@Override
|
||||
public ParseResult parse(InputStream inputStream, String filename, CsvColumnMapping mapping) {
|
||||
if (mapping == null) {
|
||||
throw new BankStatementParseException(
|
||||
"CSV parser requires a CsvColumnMapping (filename=" + filename + ")");
|
||||
}
|
||||
Charset charset = resolveCharset(mapping.getEncoding());
|
||||
char delimiter = resolveDelimiter(mapping.getDelimiter());
|
||||
int skipRows = mapping.getSkipHeaderRows() != null ? mapping.getSkipHeaderRows() : 0;
|
||||
DateTimeFormatter dateFormatter = DateTimeFormatter.ofPattern(mapping.getDateFormat());
|
||||
char decimalSep = mapping.getDecimalSeparator() != null && !mapping.getDecimalSeparator().isEmpty()
|
||||
? mapping.getDecimalSeparator().charAt(0)
|
||||
: ',';
|
||||
|
||||
List<ParsedTransaction> transactions = new ArrayList<>();
|
||||
List<String> warnings = new ArrayList<>();
|
||||
|
||||
CSVFormat format = CSVFormat.Builder.create()
|
||||
.setDelimiter(delimiter)
|
||||
.setQuote('"')
|
||||
.setIgnoreEmptyLines(true)
|
||||
.setTrim(true)
|
||||
.setAllowMissingColumnNames(true)
|
||||
.build();
|
||||
|
||||
try (Reader reader = new InputStreamReader(inputStream, charset);
|
||||
CSVParser parser = format.parse(reader)) {
|
||||
|
||||
int rowNumber = 0;
|
||||
for (CSVRecord record : parser) {
|
||||
rowNumber++;
|
||||
// Skip configurable header rows (banks vary: 0 if header-less, 1 for column
|
||||
// names, up to 5 for full metadata preambles).
|
||||
if (rowNumber <= skipRows) {
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
ParsedTransaction tx = parseRow(record, mapping, dateFormatter, decimalSep);
|
||||
if (tx != null) transactions.add(tx);
|
||||
} catch (RuntimeException e) {
|
||||
warnings.add("CSV row " + rowNumber + ": " + e.getMessage());
|
||||
}
|
||||
}
|
||||
} catch (IOException e) {
|
||||
throw new BankStatementParseException("CSV read error in " + filename, e);
|
||||
}
|
||||
|
||||
log.debug("CSV parsed: file={} mapping={} transactions={} warnings={}",
|
||||
filename, mapping.getName(), transactions.size(), warnings.size());
|
||||
// CSV exports rarely carry statement-level metadata (no IBAN/balance), so the
|
||||
// top-level fields stay null. The matching engine doesn't depend on them.
|
||||
return new ParseResult(transactions, null, null, null, null, warnings);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse one CSV record into a {@link ParsedTransaction}. Returns {@code null}
|
||||
* if the row should be silently skipped (e.g. blank required field on what
|
||||
* appears to be a non-data row like a sub-total).
|
||||
*/
|
||||
private ParsedTransaction parseRow(CSVRecord record, CsvColumnMapping mapping,
|
||||
DateTimeFormatter dateFormatter, char decimalSep) {
|
||||
// Defensive column access — banks sometimes emit short rows for sub-totals.
|
||||
String dateText = getColumn(record, mapping.getDateColumn());
|
||||
String amountText = getColumn(record, mapping.getAmountColumn());
|
||||
if (dateText == null || dateText.isEmpty() || amountText == null || amountText.isEmpty()) {
|
||||
// Likely a sub-total row or a continuation line we should ignore.
|
||||
return null;
|
||||
}
|
||||
|
||||
LocalDate bookingDate = LocalDate.parse(dateText, dateFormatter);
|
||||
int amountCents = parseAmount(amountText, decimalSep);
|
||||
|
||||
String referenceText = mapping.getReferenceColumn() != null
|
||||
? nullIfEmpty(getColumn(record, mapping.getReferenceColumn())) : null;
|
||||
String counterpartyName = mapping.getCounterpartyColumn() != null
|
||||
? nullIfEmpty(getColumn(record, mapping.getCounterpartyColumn())) : null;
|
||||
String counterpartyIban = mapping.getIbanColumn() != null
|
||||
? normalizeIban(getColumn(record, mapping.getIbanColumn())) : null;
|
||||
|
||||
// No separate value date in most CSV exports — use booking date.
|
||||
// No separate bank reference in most CSV exports.
|
||||
return new ParsedTransaction(
|
||||
bookingDate, bookingDate, amountCents, "EUR",
|
||||
referenceText, counterpartyName, counterpartyIban, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a German- or English-locale amount string to cents.
|
||||
* <p>
|
||||
* Examples (decimalSep='{@literal ,}'):
|
||||
* <ul>
|
||||
* <li>{@code "1.234,56"} → 123456</li>
|
||||
* <li>{@code "-30,00"} → -3000</li>
|
||||
* <li>{@code "100"} → 10000</li>
|
||||
* <li>{@code "0,5"} → 50</li>
|
||||
* </ul>
|
||||
* The character opposite to {@code decimalSep} is treated as the thousands
|
||||
* separator and stripped. A leading "+" is tolerated.
|
||||
*/
|
||||
static int parseAmount(String text, char decimalSep) {
|
||||
String s = text.strip();
|
||||
// Strip currency symbol or stray "+" prefix.
|
||||
if (s.startsWith("+")) s = s.substring(1);
|
||||
// The "other" separator is the thousands separator and is discarded.
|
||||
char thousandsSep = (decimalSep == ',') ? '.' : ',';
|
||||
StringBuilder cleaned = new StringBuilder(s.length());
|
||||
boolean negative = false;
|
||||
for (int i = 0; i < s.length(); i++) {
|
||||
char c = s.charAt(i);
|
||||
if (c == '-') {
|
||||
negative = true;
|
||||
} else if (c == thousandsSep || Character.isWhitespace(c)) {
|
||||
// discard
|
||||
} else if (c == decimalSep) {
|
||||
cleaned.append('.');
|
||||
} else if (Character.isDigit(c) || c == '.') {
|
||||
// After the swap above, '.' here would only occur if decimalSep is itself '.'
|
||||
cleaned.append(c);
|
||||
}
|
||||
// Any other character (currency symbol, letters) is silently dropped — they
|
||||
// appear in some exports (e.g. trailing "EUR") and are harmless.
|
||||
}
|
||||
String numericPart = cleaned.toString();
|
||||
if (numericPart.isEmpty() || ".".equals(numericPart)) {
|
||||
throw new IllegalArgumentException("amount has no digits: \"" + text + "\"");
|
||||
}
|
||||
int dotIdx = numericPart.indexOf('.');
|
||||
String euros;
|
||||
String fract;
|
||||
if (dotIdx < 0) {
|
||||
euros = numericPart;
|
||||
fract = "00";
|
||||
} else {
|
||||
euros = numericPart.substring(0, dotIdx);
|
||||
fract = numericPart.substring(dotIdx + 1);
|
||||
if (fract.length() == 1) {
|
||||
fract += "0";
|
||||
} else if (fract.length() > 2) {
|
||||
fract = fract.substring(0, 2); // truncate, never round — exact cents only
|
||||
} else if (fract.isEmpty()) {
|
||||
fract = "00";
|
||||
}
|
||||
}
|
||||
if (euros.isEmpty()) euros = "0";
|
||||
int absCents = Integer.parseInt(euros) * 100 + Integer.parseInt(fract);
|
||||
return negative ? -absCents : absCents;
|
||||
}
|
||||
|
||||
private static String getColumn(CSVRecord record, Integer idx) {
|
||||
if (idx == null || idx < 0 || idx >= record.size()) return null;
|
||||
return record.get(idx);
|
||||
}
|
||||
|
||||
private static String nullIfEmpty(String s) {
|
||||
return (s == null || s.isEmpty()) ? null : s;
|
||||
}
|
||||
|
||||
private static String normalizeIban(String raw) {
|
||||
if (raw == null) return null;
|
||||
String clean = raw.replaceAll("\\s", "").toUpperCase();
|
||||
return clean.isEmpty() ? null : clean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the encoding string from the mapping into a {@link Charset}.
|
||||
* Falls back to ISO-8859-1 on unknown/invalid values rather than failing — the
|
||||
* import session is already in progress and a slightly mangled name field is
|
||||
* preferable to a hard failure.
|
||||
*/
|
||||
private Charset resolveCharset(String name) {
|
||||
if (name == null || name.isEmpty()) return StandardCharsets.ISO_8859_1;
|
||||
try {
|
||||
return Charset.forName(name);
|
||||
} catch (IllegalCharsetNameException | UnsupportedCharsetException e) {
|
||||
log.warn("CSV: unknown encoding '{}', falling back to ISO-8859-1", name);
|
||||
return StandardCharsets.ISO_8859_1;
|
||||
}
|
||||
}
|
||||
|
||||
private char resolveDelimiter(String s) {
|
||||
if (s == null || s.isEmpty()) return ';';
|
||||
// Support escape sequence "\t" for tab — a common delimiter that's hard to type
|
||||
// in a UI form.
|
||||
if ("\\t".equals(s) || "\t".equals(s)) return '\t';
|
||||
return s.charAt(0);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,523 @@
|
||||
package de.cannamanage.service.bankimport;
|
||||
|
||||
import de.cannamanage.domain.entity.CsvColumnMapping;
|
||||
import de.cannamanage.domain.enums.BankFormat;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.LocalDate;
|
||||
import java.time.MonthDay;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Parser for SWIFT MT940 bank statements.
|
||||
* <p>
|
||||
* MT940 is a line-oriented text format used by virtually all German banks for
|
||||
* legacy account statement exports. Each "tag" starts with {@code :NN:} or
|
||||
* {@code :NNL:} (e.g. {@code :20:}, {@code :60F:}, {@code :61:}, {@code :86:}).
|
||||
* <p>
|
||||
* Tags we care about:
|
||||
* <ul>
|
||||
* <li>{@code :20:} — Transaction reference, marks start of the SWIFT block.
|
||||
* Everything before this is treated as a proprietary header and skipped.</li>
|
||||
* <li>{@code :25:} — Account identification (BLZ/account or IBAN, bank-dependent)</li>
|
||||
* <li>{@code :60F:}/{@code :60M:} — Opening balance (first/intermediate)</li>
|
||||
* <li>{@code :61:} — Statement line (one per transaction)</li>
|
||||
* <li>{@code :86:} — Information to account owner ("Verwendungszweck"), may
|
||||
* span multiple lines and carry sub-fields like {@code SVWZ+}, {@code EREF+}</li>
|
||||
* <li>{@code :62F:}/{@code :62M:} — Closing balance (final/intermediate)</li>
|
||||
* </ul>
|
||||
* <p>
|
||||
* <strong>Proprietary headers:</strong> Tools like StarMoney, WISO Mein Geld, and
|
||||
* Hibiscus often wrap the SWIFT content with proprietary header lines (version
|
||||
* markers, account exports, BOM bytes). We tolerate this by skipping everything
|
||||
* up to the first {@code :20:} tag.
|
||||
* <p>
|
||||
* <strong>Encoding:</strong> MT940 is technically a 7-bit ASCII format but German
|
||||
* banks routinely use ISO-8859-1 for umlauts in {@code :86:} fields. We decode
|
||||
* with ISO-8859-1 which is a strict superset of ASCII and safe for both cases.
|
||||
*/
|
||||
@Component
|
||||
public class Mt940Parser implements BankStatementParser {
|
||||
|
||||
private static final Logger log = LoggerFactory.getLogger(Mt940Parser.class);
|
||||
|
||||
/**
|
||||
* Century boundary for the 2-digit year in {@code :61:} dates.
|
||||
* Per German banking convention: YY in {@code [00, 70)} → 20YY, YY in {@code [70, 99]} → 19YY.
|
||||
* This handles legacy statements from the 1990s while correctly interpreting all
|
||||
* 21st-century dates up to 2069.
|
||||
*/
|
||||
static final int CENTURY_BOUNDARY = 70;
|
||||
|
||||
private static final DateTimeFormatter MMDD = DateTimeFormatter.ofPattern("MMdd");
|
||||
|
||||
/** Matches any line starting with a SWIFT tag, capturing the tag name and the payload. */
|
||||
private static final Pattern TAG_LINE = Pattern.compile("^:(\\d{2}[A-Z]?):(.*)$");
|
||||
|
||||
/**
|
||||
* Matches the entry-line ({@code :61:}) header up to and including the amount.
|
||||
* Layout: YYMMDD [MMDD entry-date] (C|D|RC|RD|EC|ED) [funds-code] amount-with-comma rest.
|
||||
*/
|
||||
private static final Pattern ENTRY_LINE = Pattern.compile(
|
||||
"^(\\d{6})" // 1: value date YYMMDD
|
||||
+ "(\\d{4})?" // 2: optional booking date MMDD
|
||||
+ "(RC|RD|EC|ED|C|D)" // 3: debit/credit indicator (longer alternatives first!)
|
||||
+ "([A-Z])?" // 4: optional funds/currency code
|
||||
+ "(\\d+(?:,\\d{0,2})?)" // 5: amount with comma (cents optional, banks sometimes omit)
|
||||
+ "(.*)$" // 6: rest (transaction type, bank reference, etc.)
|
||||
);
|
||||
|
||||
@Override
|
||||
public BankFormat getSupportedFormat() {
|
||||
return BankFormat.MT940;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean canParse(String filename, byte[] headerBytes) {
|
||||
if (headerBytes == null || headerBytes.length == 0) {
|
||||
return false;
|
||||
}
|
||||
// MT940 is 7-bit ASCII for the structural part; ISO-8859-1 is safe for any preamble.
|
||||
String head = new String(headerBytes, StandardCharsets.ISO_8859_1);
|
||||
// Strong signal: the :20: tag is mandatory and identifies the start of an MT940 block.
|
||||
// We do NOT require :60F: because some proprietary exports lack it,
|
||||
// but the combination of :20: + at least one of {:25:, :61:, :60F:} is solid.
|
||||
if (!head.contains(":20:")) {
|
||||
return false;
|
||||
}
|
||||
return head.contains(":25:") || head.contains(":61:") || head.contains(":60F:");
|
||||
}
|
||||
|
||||
@Override
|
||||
public ParseResult parse(InputStream inputStream, String filename, CsvColumnMapping mapping) {
|
||||
List<ParsedTransaction> transactions = new ArrayList<>();
|
||||
List<String> warnings = new ArrayList<>();
|
||||
|
||||
String accountIban = null;
|
||||
Integer openingBalanceCents = null;
|
||||
Integer closingBalanceCents = null;
|
||||
LocalDate statementDate = null;
|
||||
LocalDate referenceYearAnchor = null; // used to infer year for :61: entries
|
||||
|
||||
// State machine: which tag's continuation lines are we currently in?
|
||||
// MT940 tags can wrap: a :86: tag's content continues on subsequent lines that
|
||||
// don't themselves start with :NN:.
|
||||
String currentTag = null;
|
||||
StringBuilder currentPayload = new StringBuilder();
|
||||
// Entry-line state — when we close a :61: + optional :86: block, emit a transaction.
|
||||
Mt940EntryBuilder pendingEntry = null;
|
||||
boolean inSwiftBlock = false; // becomes true once we've seen the first :20:
|
||||
|
||||
try (BufferedReader reader = new BufferedReader(
|
||||
new InputStreamReader(inputStream, StandardCharsets.ISO_8859_1))) {
|
||||
|
||||
String line;
|
||||
int lineNo = 0;
|
||||
while ((line = reader.readLine()) != null) {
|
||||
lineNo++;
|
||||
// Strip trailing CR (MT940 files are often CRLF) and any leading BOM-ish noise.
|
||||
line = stripBom(line).stripTrailing();
|
||||
if (line.isEmpty() || "-".equals(line)) {
|
||||
// "-" is the SWIFT block-end marker; blank lines are decorative.
|
||||
continue;
|
||||
}
|
||||
|
||||
Matcher tagMatch = TAG_LINE.matcher(line);
|
||||
if (tagMatch.matches()) {
|
||||
// Flush the previous tag before processing this one.
|
||||
if (currentTag != null) {
|
||||
pendingEntry = handleTag(
|
||||
currentTag, currentPayload.toString(),
|
||||
transactions, warnings, pendingEntry, referenceYearAnchor);
|
||||
}
|
||||
currentTag = tagMatch.group(1);
|
||||
currentPayload.setLength(0);
|
||||
currentPayload.append(tagMatch.group(2));
|
||||
|
||||
if (!inSwiftBlock) {
|
||||
if ("20".equals(currentTag)) {
|
||||
inSwiftBlock = true;
|
||||
} else {
|
||||
// Still in the proprietary header — drop the tag and continue scanning.
|
||||
warnings.add("MT940: skipped pre-:20: line " + lineNo + " (proprietary header)");
|
||||
currentTag = null;
|
||||
currentPayload.setLength(0);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
// Side-effects we capture on tag dispatch (need data BEFORE further parsing).
|
||||
if ("25".equals(currentTag)) {
|
||||
accountIban = extractIbanFromAccount(tagMatch.group(2));
|
||||
} else if ("60F".equals(currentTag) || "60M".equals(currentTag)) {
|
||||
BalanceParse bal = parseBalanceTag(tagMatch.group(2));
|
||||
if (bal != null) {
|
||||
openingBalanceCents = bal.amountCents;
|
||||
if (bal.date != null) {
|
||||
referenceYearAnchor = bal.date;
|
||||
if (statementDate == null) {
|
||||
statementDate = bal.date;
|
||||
}
|
||||
}
|
||||
}
|
||||
} else if ("62F".equals(currentTag) || "62M".equals(currentTag)) {
|
||||
BalanceParse bal = parseBalanceTag(tagMatch.group(2));
|
||||
if (bal != null) {
|
||||
closingBalanceCents = bal.amountCents;
|
||||
if (bal.date != null) {
|
||||
statementDate = bal.date; // closing date wins
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Continuation of the current tag — append with a newline for :86:
|
||||
// (so multi-line Verwendungszweck retains breaks), space-joined otherwise.
|
||||
if (currentTag != null) {
|
||||
if ("86".equals(currentTag)) {
|
||||
currentPayload.append('\n').append(line);
|
||||
} else {
|
||||
currentPayload.append(line);
|
||||
}
|
||||
}
|
||||
// Lines outside any tag (and outside the SWIFT block) are just header noise.
|
||||
}
|
||||
}
|
||||
// Flush trailing tag.
|
||||
if (currentTag != null) {
|
||||
pendingEntry = handleTag(
|
||||
currentTag, currentPayload.toString(),
|
||||
transactions, warnings, pendingEntry, referenceYearAnchor);
|
||||
}
|
||||
// Emit a pending entry that had no :86: continuation.
|
||||
if (pendingEntry != null) {
|
||||
transactions.add(pendingEntry.build(null));
|
||||
}
|
||||
|
||||
} catch (java.io.IOException e) {
|
||||
throw new BankStatementParseException("MT940 read error in " + filename, e);
|
||||
}
|
||||
|
||||
log.debug("MT940 parsed: file={} transactions={} warnings={}",
|
||||
filename, transactions.size(), warnings.size());
|
||||
return new ParseResult(transactions, accountIban, statementDate,
|
||||
openingBalanceCents, closingBalanceCents, warnings);
|
||||
}
|
||||
|
||||
/**
|
||||
* Dispatch a fully accumulated tag to its handler. Returns the (possibly updated)
|
||||
* pending entry — {@code :61:} starts one, {@code :86:} completes it, anything else
|
||||
* resets it.
|
||||
*/
|
||||
private Mt940EntryBuilder handleTag(
|
||||
String tag, String payload,
|
||||
List<ParsedTransaction> transactions, List<String> warnings,
|
||||
Mt940EntryBuilder pendingEntry, LocalDate referenceYearAnchor) {
|
||||
switch (tag) {
|
||||
case "61" -> {
|
||||
// A new :61: starts a new entry — flush any unfinished one first.
|
||||
if (pendingEntry != null) {
|
||||
transactions.add(pendingEntry.build(null));
|
||||
}
|
||||
Mt940EntryBuilder builder = parseEntryLine(payload, referenceYearAnchor, warnings);
|
||||
return builder; // may be null if the line was unparseable
|
||||
}
|
||||
case "86" -> {
|
||||
// Completion of the most recent :61: — attach the Verwendungszweck.
|
||||
if (pendingEntry != null) {
|
||||
transactions.add(pendingEntry.build(payload));
|
||||
return null;
|
||||
}
|
||||
// :86: without preceding :61: is the "account information" tag at the file
|
||||
// level — not a transaction reference. Silently ignore.
|
||||
return null;
|
||||
}
|
||||
default -> {
|
||||
// Any other tag implicitly closes a pending entry.
|
||||
if (pendingEntry != null) {
|
||||
transactions.add(pendingEntry.build(null));
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a {@code :61:} statement-line payload. Format (positional):
|
||||
* <pre>
|
||||
* YYMMDD[MMDD](C|D|RC|RD|EC|ED)[funds]AMOUNT[rest]
|
||||
* </pre>
|
||||
* Returns {@code null} if the payload doesn't match (logged as a warning).
|
||||
*/
|
||||
private Mt940EntryBuilder parseEntryLine(String payload, LocalDate referenceYearAnchor,
|
||||
List<String> warnings) {
|
||||
Matcher m = ENTRY_LINE.matcher(payload);
|
||||
if (!m.matches()) {
|
||||
warnings.add("MT940: unparseable :61: line: " + truncate(payload, 80));
|
||||
return null;
|
||||
}
|
||||
|
||||
String valueDateRaw = m.group(1); // YYMMDD
|
||||
String bookingDateRaw = m.group(2); // MMDD or null
|
||||
String indicator = m.group(3); // C, D, RC, RD, EC, ED
|
||||
String amountRaw = m.group(5); // "123,45"
|
||||
String rest = m.group(6); // type code + bank reference
|
||||
|
||||
LocalDate valueDate = parseSwiftDate(valueDateRaw);
|
||||
LocalDate bookingDate = bookingDateRaw != null
|
||||
? inferBookingDate(bookingDateRaw, valueDate)
|
||||
: valueDate;
|
||||
|
||||
int amountCents = parseAmountToCents(amountRaw);
|
||||
// RC/RD = reversal — flips the sign vs. its base C/D indicator.
|
||||
boolean isDebit = "D".equals(indicator) || "ED".equals(indicator) || "RC".equals(indicator);
|
||||
// Note: "RC" = Reversal of a Credit = effectively a debit; "RD" = reversal of a debit = credit.
|
||||
if ("RD".equals(indicator)) isDebit = false;
|
||||
int signedAmount = isDebit ? -amountCents : amountCents;
|
||||
|
||||
String bankReference = extractBankReference(rest);
|
||||
|
||||
return new Mt940EntryBuilder(bookingDate, valueDate, signedAmount, bankReference);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a YYMMDD date applying the {@link #CENTURY_BOUNDARY} rule.
|
||||
*/
|
||||
static LocalDate parseSwiftDate(String yymmdd) {
|
||||
int yy = Integer.parseInt(yymmdd.substring(0, 2));
|
||||
int mm = Integer.parseInt(yymmdd.substring(2, 4));
|
||||
int dd = Integer.parseInt(yymmdd.substring(4, 6));
|
||||
int year = (yy >= CENTURY_BOUNDARY) ? 1900 + yy : 2000 + yy;
|
||||
return LocalDate.of(year, mm, dd);
|
||||
}
|
||||
|
||||
/**
|
||||
* Booking date is MMDD only — infer the year from the value date.
|
||||
* Most cases: same year. Edge case: value date in early January, booking date in late
|
||||
* December (or vice versa) — choose the year that puts the booking date within 30 days
|
||||
* of the value date.
|
||||
*/
|
||||
private LocalDate inferBookingDate(String mmdd, LocalDate valueDate) {
|
||||
MonthDay md = MonthDay.parse(mmdd, MMDD);
|
||||
LocalDate sameYear = md.atYear(valueDate.getYear());
|
||||
long deltaSameYear = Math.abs(sameYear.toEpochDay() - valueDate.toEpochDay());
|
||||
if (deltaSameYear <= 30) {
|
||||
return sameYear;
|
||||
}
|
||||
// Try previous and next year, pick the closer one.
|
||||
LocalDate prevYear = md.atYear(valueDate.getYear() - 1);
|
||||
LocalDate nextYear = md.atYear(valueDate.getYear() + 1);
|
||||
long deltaPrev = Math.abs(prevYear.toEpochDay() - valueDate.toEpochDay());
|
||||
long deltaNext = Math.abs(nextYear.toEpochDay() - valueDate.toEpochDay());
|
||||
return deltaPrev <= deltaNext ? prevYear : nextYear;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a SWIFT amount string ("1234,56" or "1234,5" or "1234") to cents.
|
||||
* SWIFT uses comma as decimal separator and never has a thousands separator.
|
||||
*/
|
||||
static int parseAmountToCents(String amount) {
|
||||
int commaIdx = amount.indexOf(',');
|
||||
if (commaIdx < 0) {
|
||||
return Integer.parseInt(amount) * 100;
|
||||
}
|
||||
String euros = amount.substring(0, commaIdx);
|
||||
String fract = amount.substring(commaIdx + 1);
|
||||
if (fract.isEmpty()) {
|
||||
return Integer.parseInt(euros) * 100;
|
||||
}
|
||||
if (fract.length() == 1) {
|
||||
fract += "0";
|
||||
} else if (fract.length() > 2) {
|
||||
fract = fract.substring(0, 2); // truncate, not round — banks always quote exact cents
|
||||
}
|
||||
return Integer.parseInt(euros) * 100 + Integer.parseInt(fract);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a balance tag payload ({@code :60F:}/{@code :60M:}/{@code :62F:}/{@code :62M:}).
|
||||
* Layout: (D|C) YYMMDD CCC AMOUNT.
|
||||
*/
|
||||
private BalanceParse parseBalanceTag(String payload) {
|
||||
if (payload == null || payload.length() < 10) return null;
|
||||
try {
|
||||
char sign = payload.charAt(0);
|
||||
LocalDate date = parseSwiftDate(payload.substring(1, 7));
|
||||
// Currency code is positions 7-10, we don't currently surface it on balances.
|
||||
int amountCents = parseAmountToCents(payload.substring(10));
|
||||
int signed = (sign == 'D') ? -amountCents : amountCents;
|
||||
return new BalanceParse(date, signed);
|
||||
} catch (RuntimeException e) {
|
||||
log.debug("Unparseable balance tag payload: {}", payload, e);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Heuristic IBAN extraction from a {@code :25:} payload.
|
||||
* Bank-account formats vary: some put "BLZ/account" (10-digit BLZ + slash + account),
|
||||
* others put the IBAN directly. We accept anything that looks like a German IBAN.
|
||||
*/
|
||||
private String extractIbanFromAccount(String payload) {
|
||||
if (payload == null) return null;
|
||||
String clean = payload.replaceAll("\\s", "").toUpperCase();
|
||||
// German IBAN: DE + 20 digits = 22 chars total
|
||||
Matcher m = Pattern.compile("(DE\\d{20})").matcher(clean);
|
||||
return m.find() ? m.group(1) : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the bank's own transaction reference from the trailing part of {@code :61:}.
|
||||
* Format: {@code <typeCode>//<bankRef>[\n<supplementary>]}. The typeCode is 4 chars
|
||||
* (e.g. {@code NMSC}, {@code NTRF}), the bankRef is up to 16 chars after the {@code //}.
|
||||
*/
|
||||
private String extractBankReference(String rest) {
|
||||
if (rest == null) return null;
|
||||
int slashIdx = rest.indexOf("//");
|
||||
if (slashIdx < 0) return null;
|
||||
String ref = rest.substring(slashIdx + 2);
|
||||
int newlineIdx = ref.indexOf('\n');
|
||||
if (newlineIdx > 0) ref = ref.substring(0, newlineIdx);
|
||||
ref = ref.strip();
|
||||
return ref.isEmpty() ? null : ref;
|
||||
}
|
||||
|
||||
/** Remove a leading UTF-8 BOM (0xFEFF) if present after charset decoding. */
|
||||
private String stripBom(String s) {
|
||||
if (!s.isEmpty() && s.charAt(0) == '\uFEFF') {
|
||||
return s.substring(1);
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
private static String truncate(String s, int max) {
|
||||
return s == null || s.length() <= max ? s : s.substring(0, max) + "…";
|
||||
}
|
||||
|
||||
/** Mutable builder for an in-flight :61:+:86: pair. */
|
||||
private static final class Mt940EntryBuilder {
|
||||
final LocalDate bookingDate;
|
||||
final LocalDate valueDate;
|
||||
final int amountCents;
|
||||
final String bankReference;
|
||||
|
||||
Mt940EntryBuilder(LocalDate bookingDate, LocalDate valueDate,
|
||||
int amountCents, String bankReference) {
|
||||
this.bookingDate = bookingDate;
|
||||
this.valueDate = valueDate;
|
||||
this.amountCents = amountCents;
|
||||
this.bankReference = bankReference;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the final transaction, parsing the optional :86: payload into
|
||||
* (referenceText, counterpartyName, counterpartyIban).
|
||||
*/
|
||||
ParsedTransaction build(String tag86Payload) {
|
||||
String referenceText = null;
|
||||
String counterpartyName = null;
|
||||
String counterpartyIban = null;
|
||||
|
||||
if (tag86Payload != null && !tag86Payload.isEmpty()) {
|
||||
// :86: structure varies by bank. The most common format uses {@code ?NN}
|
||||
// subfield markers (?20-?29 = Verwendungszweck, ?32/?33 = Name, ?31 = IBAN).
|
||||
// The newer SVWZ+/EREF+/CRED+/DEBT+ format (SEPA) embeds tagged values.
|
||||
Mt86Parsed parsed = parseTag86(tag86Payload);
|
||||
referenceText = parsed.referenceText;
|
||||
counterpartyName = parsed.counterpartyName;
|
||||
counterpartyIban = parsed.counterpartyIban;
|
||||
}
|
||||
|
||||
return new ParsedTransaction(
|
||||
bookingDate, valueDate, amountCents, "EUR",
|
||||
referenceText, counterpartyName, counterpartyIban, bankReference);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a multi-line :86: payload into its three semantic parts.
|
||||
* Supports both legacy {@code ?NN} subfield format and SEPA-style {@code SVWZ+}/{@code EREF+} tags.
|
||||
*/
|
||||
private static Mt86Parsed parseTag86(String payload) {
|
||||
// Normalize: collapse line breaks within the payload (subfields can wrap across lines).
|
||||
String collapsed = payload.replace("\n", "");
|
||||
|
||||
String referenceText = null;
|
||||
String counterpartyName = null;
|
||||
String counterpartyIban = null;
|
||||
|
||||
// Try the ?NN subfield format first. Pattern: ?20...?29 = Verwendungszweck (concat),
|
||||
// ?32 + ?33 = name, ?31 = counterparty IBAN, ?30 = counterparty BIC.
|
||||
if (collapsed.contains("?")) {
|
||||
StringBuilder svwz = new StringBuilder();
|
||||
StringBuilder name = new StringBuilder();
|
||||
// Split on ? but keep the marker by using a lookahead.
|
||||
String[] parts = collapsed.split("(?=\\?\\d{2})");
|
||||
for (String part : parts) {
|
||||
if (part.length() < 3 || part.charAt(0) != '?') continue;
|
||||
String key = part.substring(1, 3);
|
||||
String val = part.substring(3);
|
||||
int code;
|
||||
try { code = Integer.parseInt(key); } catch (NumberFormatException e) { continue; }
|
||||
if (code >= 20 && code <= 29) {
|
||||
svwz.append(val);
|
||||
} else if (code == 32 || code == 33) {
|
||||
name.append(val);
|
||||
} else if (code == 31) {
|
||||
counterpartyIban = val.replaceAll("\\s", "");
|
||||
}
|
||||
}
|
||||
if (svwz.length() > 0) referenceText = svwz.toString().strip();
|
||||
if (name.length() > 0) counterpartyName = name.toString().strip();
|
||||
}
|
||||
|
||||
// SEPA tagged format inside the Verwendungszweck. Extract embedded references if present.
|
||||
if (referenceText != null) {
|
||||
// Strip the inner SVWZ+ prefix if present so the visible reference is clean.
|
||||
int svwzIdx = referenceText.indexOf("SVWZ+");
|
||||
if (svwzIdx >= 0) {
|
||||
String svwzPart = referenceText.substring(svwzIdx + 5);
|
||||
int nextTagIdx = findNextSepaTag(svwzPart);
|
||||
referenceText = (nextTagIdx >= 0 ? svwzPart.substring(0, nextTagIdx) : svwzPart).strip();
|
||||
}
|
||||
} else if (collapsed.contains("SVWZ+")) {
|
||||
// No ?NN subfields, only SEPA tags.
|
||||
int svwzIdx = collapsed.indexOf("SVWZ+");
|
||||
String svwzPart = collapsed.substring(svwzIdx + 5);
|
||||
int nextTagIdx = findNextSepaTag(svwzPart);
|
||||
referenceText = (nextTagIdx >= 0 ? svwzPart.substring(0, nextTagIdx) : svwzPart).strip();
|
||||
}
|
||||
|
||||
// If nothing matched any structured format, treat the whole thing as the reference.
|
||||
if (referenceText == null && !collapsed.contains("?")) {
|
||||
referenceText = collapsed.strip();
|
||||
}
|
||||
|
||||
return new Mt86Parsed(referenceText, counterpartyName, counterpartyIban);
|
||||
}
|
||||
|
||||
/** Find the next SEPA tag (SVWZ+, EREF+, KREF+, MREF+, CRED+, DEBT+, ABWA+, ABWE+, IBAN+, BIC+). */
|
||||
private static int findNextSepaTag(String s) {
|
||||
String[] tags = {"EREF+", "KREF+", "MREF+", "CRED+", "DEBT+", "ABWA+", "ABWE+",
|
||||
"IBAN+", "BIC+", "SVWZ+"};
|
||||
int earliest = -1;
|
||||
for (String t : tags) {
|
||||
int idx = s.indexOf(t);
|
||||
if (idx >= 0 && (earliest < 0 || idx < earliest)) earliest = idx;
|
||||
}
|
||||
return earliest;
|
||||
}
|
||||
|
||||
private record Mt86Parsed(String referenceText, String counterpartyName, String counterpartyIban) {}
|
||||
|
||||
private record BalanceParse(LocalDate date, int amountCents) {}
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
package de.cannamanage.service.bankimport;
|
||||
|
||||
import java.time.LocalDate;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Aggregated result of parsing one bank statement file.
|
||||
* <p>
|
||||
* Carries the parsed transactions plus statement-level metadata (account IBAN,
|
||||
* opening/closing balances) which is used for downstream sanity checks
|
||||
* (e.g. verifying that "sum of transactions" matches "closing − opening").
|
||||
* <p>
|
||||
* {@link #warnings} captures non-fatal parsing issues — fields that were missing,
|
||||
* lines that were skipped, unexpected format variants. These are surfaced in the
|
||||
* UI but don't abort the import.
|
||||
*
|
||||
* @param transactions parsed transactions, in source-file order
|
||||
* @param accountIban account this statement belongs to (may be {@code null} for CSV)
|
||||
* @param statementDate end date of the statement period (or transaction date for single-line statements)
|
||||
* @param openingBalanceCents opening balance in cents — signed, optional
|
||||
* @param closingBalanceCents closing balance in cents — signed, optional
|
||||
* @param warnings non-fatal parser warnings; empty list if clean
|
||||
*/
|
||||
public record ParseResult(
|
||||
List<ParsedTransaction> transactions,
|
||||
String accountIban,
|
||||
LocalDate statementDate,
|
||||
Integer openingBalanceCents,
|
||||
Integer closingBalanceCents,
|
||||
List<String> warnings
|
||||
) {}
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
package de.cannamanage.service.bankimport;
|
||||
|
||||
import java.time.LocalDate;
|
||||
|
||||
/**
|
||||
* Sprint 10 — Single transaction extracted from a bank statement by a {@link BankStatementParser}.
|
||||
* <p>
|
||||
* Sign convention for {@link #amountCents}: <strong>positive = incoming</strong>
|
||||
* (potential member payment), <strong>negative = outgoing</strong> (expense).
|
||||
* <p>
|
||||
* This is an in-memory parsing artifact only; the import orchestrator later
|
||||
* converts each instance into a persisted {@code BankTransaction} entity.
|
||||
*
|
||||
* @param bookingDate date the bank posted the transaction
|
||||
* @param valueDate date the funds became available (may equal bookingDate)
|
||||
* @param amountCents transaction amount in cents (signed)
|
||||
* @param currency ISO 4217 code, almost always {@code "EUR"} for German banks
|
||||
* @param referenceText "Verwendungszweck" — concatenated free-text reference fields
|
||||
* @param counterpartyName payer (for credits) or payee (for debits) name
|
||||
* @param counterpartyIban payer/payee IBAN if available
|
||||
* @param bankReference bank's own transaction reference (EREF/KREF/MREF for SEPA, NtryRef for CAMT)
|
||||
*/
|
||||
public record ParsedTransaction(
|
||||
LocalDate bookingDate,
|
||||
LocalDate valueDate,
|
||||
int amountCents,
|
||||
String currency,
|
||||
String referenceText,
|
||||
String counterpartyName,
|
||||
String counterpartyIban,
|
||||
String bankReference
|
||||
) {}
|
||||
+1
-1
@@ -165,7 +165,7 @@ public class MemberListRegistryGenerator implements ReportGenerator<DateRangeRep
|
||||
return switch (status) {
|
||||
case ACTIVE -> "Aktiv";
|
||||
case SUSPENDED -> "Gesperrt";
|
||||
case RESIGNED -> "Ausgetreten";
|
||||
case RESIGNED, LEFT -> "Ausgetreten";
|
||||
case EXPELLED -> "Ausgeschlossen";
|
||||
case PENDING_APPROVAL -> "Aufnahme ausstehend";
|
||||
};
|
||||
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
package de.cannamanage.service.repository;
|
||||
|
||||
import de.cannamanage.domain.entity.BankImportSession;
|
||||
import de.cannamanage.domain.enums.ImportSessionStatus;
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Repository for {@link BankImportSession} (Sprint 10).
|
||||
* <p>
|
||||
* All queries are tenant-scoped automatically by Hibernate's {@code @Filter}
|
||||
* defined on {@code AbstractTenantEntity}, so {@code clubId} parameters are
|
||||
* an additional safety filter within a tenant rather than a security boundary.
|
||||
*/
|
||||
@Repository
|
||||
public interface BankImportSessionRepository extends JpaRepository<BankImportSession, UUID> {
|
||||
|
||||
/** Import history for a club, most recent first — drives the import history page. */
|
||||
List<BankImportSession> findByClubIdOrderByCreatedAtDesc(UUID clubId);
|
||||
|
||||
/** Used by the "Resume Import" banner: list of sessions that need finishing. */
|
||||
List<BankImportSession> findByClubIdAndStatusOrderByCreatedAtDesc(UUID clubId, ImportSessionStatus status);
|
||||
|
||||
/**
|
||||
* Duplicate-import guard: returns a recent session for this club with the same filename,
|
||||
* created after the given cutoff. Used to warn the admin before re-importing the same file.
|
||||
*/
|
||||
Optional<BankImportSession> findFirstByClubIdAndFilenameAndCreatedAtAfter(
|
||||
UUID clubId, String filename, Instant cutoff);
|
||||
|
||||
/** Tier-limit enforcement: count Starter-plan imports in the current month. */
|
||||
long countByClubIdAndCreatedAtAfter(UUID clubId, Instant since);
|
||||
}
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
package de.cannamanage.service.repository;
|
||||
|
||||
import de.cannamanage.domain.entity.BankTransaction;
|
||||
import de.cannamanage.domain.enums.MatchStatus;
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Repository for {@link BankTransaction} (Sprint 10).
|
||||
* <p>
|
||||
* Tenant-scoping is applied automatically by Hibernate {@code @Filter}.
|
||||
*/
|
||||
@Repository
|
||||
public interface BankTransactionRepository extends JpaRepository<BankTransaction, UUID> {
|
||||
|
||||
/** All transactions of a session, in booking order — drives the review table. */
|
||||
List<BankTransaction> findBySessionIdOrderByBookingDateAsc(UUID sessionId);
|
||||
|
||||
/** Filter by review status — for the "Matched / Suggested / Unmatched / Skipped" tabs. */
|
||||
List<BankTransaction> findBySessionIdAndMatchStatus(UUID sessionId, MatchStatus matchStatus);
|
||||
|
||||
/** Counters for session-level statistics displayed in the wizard. */
|
||||
long countBySessionIdAndMatchStatus(UUID sessionId, MatchStatus matchStatus);
|
||||
}
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
package de.cannamanage.service.repository;
|
||||
|
||||
import de.cannamanage.domain.entity.CsvColumnMapping;
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
import java.util.UUID;
|
||||
|
||||
/**
|
||||
* Repository for {@link CsvColumnMapping} (Sprint 10).
|
||||
* <p>
|
||||
* Tenant-scoping is applied automatically by Hibernate {@code @Filter}.
|
||||
*/
|
||||
@Repository
|
||||
public interface CsvColumnMappingRepository extends JpaRepository<CsvColumnMapping, UUID> {
|
||||
|
||||
/** All saved CSV mappings for a club — drives the template dropdown in the upload wizard. */
|
||||
List<CsvColumnMapping> findByClubId(UUID clubId);
|
||||
|
||||
/** Default mapping (if any) — pre-selected in the upload wizard. */
|
||||
Optional<CsvColumnMapping> findByClubIdAndIsDefaultTrue(UUID clubId);
|
||||
|
||||
/** Tier-limit enforcement: Pro plan = max 3 templates per club. */
|
||||
long countByClubId(UUID clubId);
|
||||
}
|
||||
@@ -0,0 +1,339 @@
|
||||
# Sprint 10 Analysis — Smart Payment Import (Kontoauszug-Abgleich)
|
||||
|
||||
**Date:** 2026-06-15
|
||||
**Author:** Patrick Plate / Lumen (Architect)
|
||||
**Status:** Draft v1
|
||||
**Sprint Goal:** Automated bank statement import with intelligent payment matching
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem Statement
|
||||
|
||||
Cannabis clubs (Anbauvereinigungen) collect monthly/quarterly member fees. Currently, the admin must manually record each payment in the system — comparing bank statements line-by-line against expected member fees. For a club with 200 members making monthly payments, this means 200+ manual Payment entries per month.
|
||||
|
||||
**Pain points:**
|
||||
- Time-consuming: ~2-4 hours/month for 200-member clubs
|
||||
- Error-prone: manual transcription leads to mismatches
|
||||
- Delayed bookkeeping: payments sit unrecorded for days/weeks
|
||||
- No audit trail of the import source (original bank file)
|
||||
- Difficult to identify missing payments (overdue detection lags)
|
||||
|
||||
**Opportunity:** German banks universally support MT940 (SWIFT standard) and increasingly CAMT.053 (ISO 20022 XML) for electronic bank statement export. Automating the import and matching process reduces the workload from hours to minutes.
|
||||
|
||||
---
|
||||
|
||||
## 2. Affected Components
|
||||
|
||||
### Existing (Sprint 8 — Finance Module)
|
||||
|
||||
| Component | Path | Role |
|
||||
|-----------|------|------|
|
||||
| `FinanceService` | `cannamanage-service/.../service/FinanceService.java` | Payment recording, ledger management |
|
||||
| `Payment` entity | `cannamanage-domain/.../entity/Payment.java` | Payment records with member + period |
|
||||
| `LedgerEntry` entity | `cannamanage-domain/.../entity/LedgerEntry.java` | Double-entry financial journal |
|
||||
| `FeeSchedule` | `cannamanage-domain/.../entity/FeeSchedule.java` | Fee definitions (amount, interval) |
|
||||
| `MemberFeeAssignment` | `cannamanage-domain/.../entity/MemberFeeAssignment.java` | Member ↔ Fee schedule link |
|
||||
| `PaymentMethod` enum | `cannamanage-domain/.../enums/PaymentMethod.java` | CASH, BANK_TRANSFER, SEPA_LASTSCHRIFT, OTHER |
|
||||
| `finance.ts` service | `cannamanage-frontend/src/services/finance.ts` | Frontend API hooks for finance |
|
||||
| Finance pages | `cannamanage-frontend/src/app/.../finance/` | Admin finance management UI |
|
||||
|
||||
### New (Sprint 10)
|
||||
|
||||
| Component | Purpose |
|
||||
|-----------|---------|
|
||||
| `BankImportSession` entity | Tracks each uploaded file + processing status |
|
||||
| `BankTransaction` entity | Individual parsed transactions from bank file |
|
||||
| `CsvColumnMapping` entity | Saved CSV column templates per bank |
|
||||
| `BankStatementParserService` | Format detection + delegation to format-specific parsers |
|
||||
| `Mt940Parser` | SWIFT MT940 text format parser |
|
||||
| `Camt053Parser` | ISO 20022 XML parser |
|
||||
| `CsvBankParser` | Configurable CSV parser |
|
||||
| `PaymentMatchingService` | Weighted matching algorithm with confidence scoring |
|
||||
| `BankImportService` | Orchestration: upload → parse → match → confirm |
|
||||
| Import UI wizard | 4-step frontend wizard for the import flow |
|
||||
|
||||
---
|
||||
|
||||
## 3. Current State (Ist-Zustand)
|
||||
|
||||
### Finance Module (Sprint 8)
|
||||
|
||||
The finance module already supports:
|
||||
- **Fee Schedules:** Define fee amounts + intervals (MONTHLY, QUARTERLY, YEARLY, ONE_TIME)
|
||||
- **Member Fee Assignments:** Link members to fee schedules with validity periods
|
||||
- **Payment Recording:** Manual payment entry with member, amount, method, period, reference
|
||||
- **Ledger Entries:** Automatic INCOME entry on payment, EXPENSE entries for club costs
|
||||
- **Member Balance:** Calculated outstanding amounts per member
|
||||
- **Notifications:** PAYMENT_REMINDER, PAYMENT_OVERDUE, PAYMENT_RECEIVED types exist
|
||||
- **Receipts:** Auto-generated on payment confirmation
|
||||
|
||||
### What's Missing
|
||||
|
||||
- No file upload capability for bank statements
|
||||
- No MT940/CAMT.053 parsing
|
||||
- No automatic matching logic
|
||||
- No import session tracking / audit trail
|
||||
- No member IBAN storage (for enhanced matching)
|
||||
- No CSV column mapping configuration
|
||||
- No GDPR consent type for bank data / IBAN storage
|
||||
- No bulk payment confirmation workflow
|
||||
|
||||
---
|
||||
|
||||
## 4. Format Specifications
|
||||
|
||||
### 4.1 MT940 (SWIFT Standard)
|
||||
|
||||
The most common bank statement export format from German banks (Sparkasse, Volksbank, Deutsche Bank, Commerzbank, ING DiBa, DKB).
|
||||
|
||||
**Structure:**
|
||||
```
|
||||
:20:TRANSACTION_REF -- Transaction reference number
|
||||
:25:BLZKTO/IBAN -- Account identification (BLZ + Account or IBAN)
|
||||
:28C:STATEMENT/PAGE -- Statement number / sequence
|
||||
:60F:D/CYYMMDDCURRENCY AMT -- Opening balance (D=debit, C=credit)
|
||||
:61:YYMMDDYYMMDDCD AMOUNT -- Transaction line (value date, booking date, D/C, amount)
|
||||
:86:PURPOSE TEXT -- Multi-line Verwendungszweck (up to 6 lines of 27 chars)
|
||||
:62F:D/CYYMMDDCURRENCY AMT -- Closing balance
|
||||
:64:D/CYYMMDDCURRENCY AMT -- Available balance
|
||||
```
|
||||
|
||||
**Key parsing challenges:**
|
||||
- Multi-line `:86:` field contains the Verwendungszweck (payment reference)
|
||||
- Amount uses comma as decimal separator (German: `1234,56`)
|
||||
- Date format is YYMMDD (2-digit year)
|
||||
- `D` = debit (outgoing), `C` = credit (incoming)
|
||||
- Some banks prefix IBAN in `:25:`, others use BLZ + account number
|
||||
- Field `:61:` encodes transaction type codes (N = normal, S = SEPA, etc.)
|
||||
|
||||
**Example transaction block:**
|
||||
```
|
||||
:61:2506150615CR50,00NMSCNONREF
|
||||
:86:SVWZ+M-0042 Mitgliedsbeitrag Juni
|
||||
EREF+NOTPROVIDED
|
||||
KREF+NOTPROVIDED
|
||||
MREF+NOTPROVIDED
|
||||
CRED+DE98ZZZ09999999999
|
||||
DEBT+DE89370400440532013000
|
||||
```
|
||||
|
||||
### 4.2 CAMT.053 (ISO 20022 XML)
|
||||
|
||||
Newer XML-based format, richer structured data. Increasingly used by German banks alongside MT940.
|
||||
|
||||
**Structure (simplified):**
|
||||
```xml
|
||||
<BkToCstmrStmt>
|
||||
<Stmt>
|
||||
<Id>Statement-ID</Id>
|
||||
<Acct><Id><IBAN>DE89370400440532013000</IBAN></Id></Acct>
|
||||
<Bal><!-- Opening/Closing balances --></Bal>
|
||||
<Ntry>
|
||||
<Amt Ccy="EUR">50.00</Amt>
|
||||
<CdtDbtInd>CRDT</CdtDbtInd> <!-- CRDT=incoming, DBIT=outgoing -->
|
||||
<BookgDt><Dt>2025-06-15</Dt></BookgDt>
|
||||
<ValDt><Dt>2025-06-15</Dt></ValDt>
|
||||
<NtryDtls>
|
||||
<TxDtls>
|
||||
<RmtInf>
|
||||
<Ustrd>M-0042 Mitgliedsbeitrag Juni</Ustrd> <!-- Verwendungszweck -->
|
||||
</RmtInf>
|
||||
<RltdPties>
|
||||
<Dbtr><Nm>Max Mustermann</Nm></Dbtr>
|
||||
<DbtrAcct><Id><IBAN>DE12345678901234567890</IBAN></Id></DbtrAcct>
|
||||
</RltdPties>
|
||||
</TxDtls>
|
||||
</NtryDtls>
|
||||
</Ntry>
|
||||
</Stmt>
|
||||
</BkToCstmrStmt>
|
||||
```
|
||||
|
||||
**Advantages over MT940:**
|
||||
- Full counterparty IBAN always available
|
||||
- Counterparty name in structured field
|
||||
- Verwendungszweck clearly separated (Ustrd = unstructured, Strd = structured)
|
||||
- ISO date format (YYYY-MM-DD)
|
||||
- Amount with dot decimal separator
|
||||
- Richer metadata (end-to-end reference, mandate ID for SEPA)
|
||||
|
||||
### 4.3 Generic CSV
|
||||
|
||||
Every German bank offers CSV download but with wildly different column layouts:
|
||||
|
||||
| Bank | Date Col | Amount Col | Reference Col | Separator | Encoding |
|
||||
|------|----------|-----------|---------------|-----------|----------|
|
||||
| Sparkasse | 0 | 4 | 8 | `;` | ISO-8859-1 |
|
||||
| ING DiBa | 0 | 7 | 4 | `;` | ISO-8859-1 |
|
||||
| DKB | 0 | 7 | 3 | `;` | ISO-8859-1 |
|
||||
| Commerzbank | 0 | 4 | 3 | `;` | UTF-8 |
|
||||
| Volksbank | 0 | 11 | 8 | `;` | ISO-8859-1 |
|
||||
|
||||
**Common German CSV traits:**
|
||||
- Semicolon delimiter (not comma)
|
||||
- ISO-8859-1 encoding (not UTF-8) for most banks
|
||||
- German date format: dd.MM.yyyy
|
||||
- German number format: `1.234,56` (dot as thousands separator, comma as decimal)
|
||||
- Header rows (1-2 lines to skip)
|
||||
- Quote character: `"` for fields containing semicolons
|
||||
|
||||
---
|
||||
|
||||
## 5. Matching Algorithm Design
|
||||
|
||||
### 5.1 Candidate Generation
|
||||
|
||||
For each incoming transaction (credit only — debits are expenses):
|
||||
1. Filter members with active fee assignments in the club
|
||||
2. Calculate expected payment amount for current/recent periods
|
||||
3. Generate candidate pairs: (transaction, member)
|
||||
|
||||
### 5.2 Weighted Scoring
|
||||
|
||||
| Criterion | Weight | Score Logic |
|
||||
|-----------|--------|-------------|
|
||||
| **Amount match** | 35% | 100 if exact match; 80 if within ±5%; 0 otherwise |
|
||||
| **Verwendungszweck contains member number** | 30% | 100 if "M-XXXX" found; 50 if partial match |
|
||||
| **Verwendungszweck contains member name** | 15% | 100 if full name match; 70 if last name only; 50 if fuzzy (Levenshtein ≤2) |
|
||||
| **IBAN match** | 15% | 100 if exact IBAN match (requires stored IBAN) |
|
||||
| **Date within payment window** | 5% | 100 if within expected period; 50 if ±30 days; 0 otherwise |
|
||||
|
||||
### 5.3 Confidence Thresholds
|
||||
|
||||
| Confidence | Classification | Action |
|
||||
|------------|---------------|--------|
|
||||
| ≥90% | AUTO_MATCHED | Green badge, ready for bulk confirm |
|
||||
| 60–89% | SUGGESTED | Yellow badge, needs manual review |
|
||||
| <60% | UNMATCHED | Red badge, manual assignment required |
|
||||
|
||||
### 5.4 Conflict Resolution
|
||||
|
||||
- If multiple members match a single transaction: pick highest confidence, mark as SUGGESTED (never AUTO)
|
||||
- If a member matches multiple transactions: possible double payment — flag for review
|
||||
- Negative amounts (outgoing): skip matching, offer expense categorization
|
||||
|
||||
---
|
||||
|
||||
## 6. Legal & Compliance Analysis
|
||||
|
||||
### 6.1 DSGVO (GDPR)
|
||||
|
||||
| Aspect | Legal Basis | Implementation |
|
||||
|--------|-------------|----------------|
|
||||
| IBAN storage | Art. 6(1)(b) — contract performance + Art. 6(1)(a) — explicit consent | New `BANK_DATA` consent type, opt-in per member |
|
||||
| Bank statement data | Art. 6(1)(f) — legitimate interest (bookkeeping) | Process & match, don't store raw file permanently |
|
||||
| Counterparty names | Art. 6(1)(f) — legitimate interest | Only display during import review, don't persist for non-members |
|
||||
| Data minimization | Art. 5(1)(c) | Delete raw import file after 30 days, keep only parsed transactions |
|
||||
| Right to erasure | Art. 17 | bank_transactions linked to member — cascade on member deletion |
|
||||
|
||||
### 6.2 Financial Retention (§147 AO / GoBD)
|
||||
|
||||
| Data | Retention Period | Basis |
|
||||
|------|-----------------|-------|
|
||||
| Bank import sessions | 10 years | §147 Abs. 1 Nr. 5 AO (Buchungsbelege) |
|
||||
| Bank transactions (parsed) | 10 years | §147 Abs. 1 Nr. 5 AO |
|
||||
| Confirmed payments | 10 years | §147 Abs. 1 Nr. 5 AO |
|
||||
| Raw import files | 30 days (then delete) | Data minimization — parsed data is the Beleg |
|
||||
| CSV column mappings | Until club deletion | Configuration data, no retention requirement |
|
||||
|
||||
### 6.3 Tier Restrictions
|
||||
|
||||
| Feature | Starter | Pro | Enterprise |
|
||||
|---------|---------|-----|-----------|
|
||||
| Imports per month | 1 | Unlimited | Unlimited |
|
||||
| Formats | CSV only | MT940 + CAMT.053 + CSV | All formats |
|
||||
| Saved column templates | 0 | 3 | Unlimited |
|
||||
| Auto-confirm (≥90%) | ❌ | ❌ | ✅ |
|
||||
| Import history retention | 3 months | 12 months | Unlimited |
|
||||
|
||||
---
|
||||
|
||||
## 7. Technology Decisions
|
||||
|
||||
### 7.1 Parser Libraries
|
||||
|
||||
| Format | Approach | Rationale |
|
||||
|--------|----------|-----------|
|
||||
| MT940 | Custom parser (no external lib) | Format is simple enough, no mature Java MT940 lib exists for Spring Boot 3.x. Hand-rolled parser gives full control over German bank quirks. |
|
||||
| CAMT.053 | JAXB with XSD-generated classes | ISO 20022 has official XSD schemas. JAXB (Jakarta XML Binding) generates type-safe classes. Already used in PAISY for GKV data exchange. |
|
||||
| CSV | Apache Commons CSV | Already a pattern in the project (semicolon + ISO-8859-1). Configurable delimiter, quote, encoding. |
|
||||
|
||||
### 7.2 File Upload
|
||||
|
||||
- Spring Boot multipart upload (`MultipartFile`)
|
||||
- Max file size: 10 MB (sufficient for years of bank statements)
|
||||
- Temporary storage: server filesystem during processing, then delete
|
||||
- No S3/cloud storage needed — parsed data is persisted in DB
|
||||
|
||||
### 7.3 Fuzzy Matching
|
||||
|
||||
- Levenshtein distance for name matching: Apache Commons Text `StringUtils.getLevenshteinDistance()`
|
||||
- Already available via Spring Boot starter dependencies
|
||||
- No external NLP/AI needed — rule-based matching is sufficient for this domain
|
||||
|
||||
---
|
||||
|
||||
## 8. Risk Assessment
|
||||
|
||||
| Risk | Probability | Impact | Mitigation |
|
||||
|------|-------------|--------|------------|
|
||||
| MT940 format variations across banks | High | Medium | Extensive test fixtures from multiple banks; graceful degradation on parse errors |
|
||||
| False positive matches (wrong member) | Medium | High | Never auto-confirm without admin review; require ≥90% confidence for green badge |
|
||||
| Performance with large files (5000+ transactions) | Low | Medium | Stream-based parsing; batch DB inserts (flush/clear every 100); async processing for large files |
|
||||
| IBAN storage GDPR complaints | Low | Medium | Explicit opt-in consent; easy deletion; clear privacy notice |
|
||||
| Duplicate import (same file uploaded twice) | Medium | Low | Detect by filename + date range + total; warn but allow (idempotent match re-run) |
|
||||
| CSV encoding issues (mojibake) | Medium | Low | Default ISO-8859-1; allow encoding override in mapping template |
|
||||
|
||||
---
|
||||
|
||||
## 9. Data Flow
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Admin uploads bank file] --> B[Format Detection]
|
||||
B -->|MT940| C[Mt940Parser]
|
||||
B -->|CAMT.053| D[Camt053Parser]
|
||||
B -->|CSV| E[CsvBankParser]
|
||||
C --> F[List of BankTransaction entities]
|
||||
D --> F
|
||||
E --> F
|
||||
F --> G[PaymentMatchingService]
|
||||
G --> H{Confidence Score}
|
||||
H -->|>= 90%| I[AUTO_MATCHED - green]
|
||||
H -->|60-89%| J[SUGGESTED - yellow]
|
||||
H -->|< 60%| K[UNMATCHED - red]
|
||||
I --> L[Admin reviews match table]
|
||||
J --> L
|
||||
K --> L
|
||||
L --> M[Confirm matches]
|
||||
M --> N[Create Payment + LedgerEntry]
|
||||
N --> O[Send PAYMENT_RECEIVED notification]
|
||||
M --> P[Categorize as expense]
|
||||
P --> Q[Create expense LedgerEntry]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Open Questions
|
||||
|
||||
- [ ] Should we support MT940 multi-statement files (multiple accounts in one file)?
|
||||
- **Recommendation:** Yes — filter by club's own IBAN, ignore other accounts.
|
||||
- [ ] Should debit transactions (outgoing) be auto-categorized as expenses?
|
||||
- **Recommendation:** Offer optional categorization, but don't auto-create. Admin selects category.
|
||||
- [ ] Should we generate a PDF reconciliation report after import completion?
|
||||
- **Recommendation:** Nice-to-have for Sprint 11. Focus on core import/match flow first.
|
||||
- [ ] Should the matching algorithm learn from previous confirmations (ML)?
|
||||
- **Recommendation:** Out of scope. Rule-based matching is sufficient for club-size data (50-500 members).
|
||||
|
||||
---
|
||||
|
||||
## 11. Integration Points
|
||||
|
||||
| System | Direction | What |
|
||||
|--------|-----------|------|
|
||||
| FinanceService | Outgoing | `recordPayment()` called for each confirmed match |
|
||||
| NotificationDispatchService | Outgoing | PAYMENT_RECEIVED to member on confirmation |
|
||||
| AuditService | Outgoing | BANK_IMPORT_STARTED, BANK_IMPORT_COMPLETED events |
|
||||
| RetentionService | Scheduled | Auto-delete raw files after 30 days; retain parsed data 10 years |
|
||||
| ConsentService | Read | Check BANK_DATA consent before storing member IBAN |
|
||||
| TierLimitService | Read | Enforce import count + format restrictions per plan tier |
|
||||
@@ -0,0 +1,234 @@
|
||||
# Sprint 10 Plan Review v3 — 6-Expert Panel (Final)
|
||||
|
||||
**Date:** 2026-06-15
|
||||
**Reviewer:** Lumen (Architect) — Multi-Expert Panel
|
||||
**Documents reviewed:**
|
||||
- `cannamanage-sprint10-analysis.md` v1
|
||||
- `cannamanage-sprint10-plan.md` v3
|
||||
- `cannamanage-sprint10-testplan.md` v1
|
||||
|
||||
**Verdict:** ✅ APPROVED
|
||||
**Panel Confidence:** 99% (exceeds 95% threshold)
|
||||
|
||||
---
|
||||
|
||||
## Changes v2 → v3
|
||||
|
||||
7 of 12 info items from the v2 review were incorporated into plan v3. The remaining 5 were deliberately excluded (either already correct as-is, deferred to Sprint 11, or not applicable).
|
||||
|
||||
| # | Info Item | Source | Resolution in v3 |
|
||||
|---|-----------|--------|-------------------|
|
||||
| 1 | Century boundary constant for MT940 dates | Domain #1 | ✅ Added `CENTURY_BOUNDARY = 70` constant with Javadoc explaining standard banking convention |
|
||||
| 2 | Skip proprietary headers before first `:20:` tag | Domain #2 | ✅ Added pre-parse header stripping for StarMoney/WISO/Hibiscus portal exports |
|
||||
| 3 | Duplicate session detection | Architecture #2 | ✅ Added `checkDuplicateImport()` method — checks filename+club within 24h, returns 409 Conflict with existing sessionId for frontend confirmation dialog |
|
||||
| 4 | Double-payment scenario handling | Testing #2 | ✅ Added explicit logic: if same member matches 2+ transactions in one file, all are downgraded to SUGGESTED status regardless of individual confidence score |
|
||||
| 5 | Constructive tier restriction messaging | UX #2 | ✅ Upload step now specifies helpful guidance text with alternative suggestions and "Plan vergleichen" link, never punitive language |
|
||||
| 6 | Fee schedule validity date context | Integration #2 | ✅ `precomputeFeeAmounts()` now accepts `bookingDateContext` parameter — queries fee assignments valid at the transaction date, not today's date |
|
||||
| 7 | Session immutability after completion (GoBD) | GoBD | ✅ Added `assertSessionMutable()` guard called by all mutation endpoints. Throws `IllegalStateException` for COMPLETED sessions. Explicit GoBD compliance. |
|
||||
|
||||
**Deliberately excluded (correct as-is or deferred):**
|
||||
|
||||
| # | Info Item | Reason for exclusion |
|
||||
|---|-----------|---------------------|
|
||||
| 8 | CsvBankParser interface deviation | Already acknowledged as acceptable in v2 review — explicit delegation pattern |
|
||||
| 9 | Performance test for 5000+ transactions | Explicitly Sprint 11 scope — production observability (timing log) is sufficient for Sprint 10 |
|
||||
| 10 | Payment void → FK behavior | Already correct as-is per v2 review — ON DELETE SET NULL handles it |
|
||||
| 11 | Mobile horizontal scroll | Acceptable for admin-only feature — not a user-facing portal page |
|
||||
| 12 | Running totals already in v2 | Already implemented in v2, reviewer noted it was addressed |
|
||||
|
||||
---
|
||||
|
||||
## Panel Composition
|
||||
|
||||
| # | Expert | Perspective | Focus Area |
|
||||
|---|--------|-------------|------------|
|
||||
| 1 | 🏛️ Domain Expert (German Vereinsrecht + Finance) | Legal compliance, financial regulations, German banking standards | §147 AO, DSGVO, MT940/CAMT standards, GoBD |
|
||||
| 2 | 🔧 Architecture Expert | System design, patterns, scalability | Service decomposition, data model, API design |
|
||||
| 3 | 🛡️ Security & Privacy Expert | Data protection, input validation, attack surface | GDPR, file upload security, IBAN handling |
|
||||
| 4 | 🧪 Testing Expert | Test coverage, edge cases, quality assurance | Test completeness, fixture quality, E2E coverage |
|
||||
| 5 | 💼 UX/Product Expert | User workflows, accessibility, error states | Import wizard UX, error recovery, progressive disclosure |
|
||||
| 6 | ⚙️ Integration Expert | System integration, backward compatibility, performance | Existing finance module, notifications, retention |
|
||||
|
||||
---
|
||||
|
||||
## Expert 1: 🏛️ Domain Expert (German Vereinsrecht + Finance)
|
||||
|
||||
**Confidence:** 99%
|
||||
|
||||
### Verdict: ✅ APPROVED
|
||||
|
||||
### Strengths (carried from v2)
|
||||
- **§147 AO compliance** correctly identified: 10-year retention for Buchungsbelege.
|
||||
- **GoBD compliance** preserved: append-only LedgerEntry pattern continues.
|
||||
- **DSGVO Art. 6(1)(a)** correctly applied for IBAN storage with explicit consent type.
|
||||
- **MT940/CAMT.053/CSV format knowledge** is accurate and comprehensive.
|
||||
|
||||
### v3 Improvements Assessed
|
||||
- **Century boundary constant** (✅): `CENTURY_BOUNDARY = 70` is the standard SWIFT convention. Documenting it with a named constant prevents future developer confusion. Good practice.
|
||||
- **Proprietary header skipping** (✅): Real-world necessity — StarMoney, WISO Mein Geld, and Hibiscus all add proprietary headers. Without this, users get parse failures and blame the software. Excellent robustness improvement.
|
||||
- **Session immutability** (✅): The explicit `assertSessionMutable()` guard makes GoBD compliance programmatically enforced rather than implicit. This is defense-in-depth for Unveränderbarkeit — an auditor can now point to a specific code guard.
|
||||
|
||||
### Findings
|
||||
|
||||
| # | Severity | Finding | Recommendation |
|
||||
|---|----------|---------|----------------|
|
||||
| — | — | No findings. All v2 info items resolved or correctly excluded. | — |
|
||||
|
||||
---
|
||||
|
||||
## Expert 2: 🔧 Architecture Expert
|
||||
|
||||
**Confidence:** 99%
|
||||
|
||||
### Verdict: ✅ APPROVED
|
||||
|
||||
### Strengths (carried from v2)
|
||||
- Strategy pattern, service decomposition, transaction boundary discipline, batch persistence all remain excellent.
|
||||
|
||||
### v3 Improvements Assessed
|
||||
- **Duplicate detection** (✅): `checkDuplicateImport()` is a clean, focused guard with 409 Conflict response. Frontend can show a confirmation dialog — standard UX pattern for "are you sure?" without blocking power users who intentionally re-import.
|
||||
- **Fee schedule validity date** (✅): Critical correctness fix. A January import reviewing December transactions *must* use December's fee schedule. The `bookingDateContext` parameter makes this explicit in the method signature — impossible to forget during implementation.
|
||||
- **Double-payment downgrade** (✅): Correct safety-first approach. If member M appears in 2 transactions, both become SUGGESTED regardless of individual confidence. The admin sees both side-by-side and can confirm intentional double payments (e.g., quarterly + one-off). No false-negative risk.
|
||||
|
||||
### Findings
|
||||
|
||||
| # | Severity | Finding | Recommendation |
|
||||
|---|----------|---------|----------------|
|
||||
| — | — | No findings. Architecture is clean and complete. | — |
|
||||
|
||||
---
|
||||
|
||||
## Expert 3: 🛡️ Security & Privacy Expert
|
||||
|
||||
**Confidence:** 99%
|
||||
|
||||
### Verdict: ✅ APPROVED
|
||||
|
||||
### Strengths (carried from v2)
|
||||
- XXE prevention, filename sanitization, permission model, tenant isolation, IBAN consent gating, rate limiting all remain solid.
|
||||
|
||||
### v3 Improvements Assessed
|
||||
- **Session immutability guard** (✅): Prevents a class of bugs where completed sessions could be accidentally mutated through direct API calls (e.g., replay attacks, browser back-button + resubmit). Good security hardening beyond just GoBD.
|
||||
- **Duplicate detection** (✅): The 409 response with existing sessionId doesn't leak sensitive data — it only confirms that a filename was used before (the admin already uploaded it, so they know). Safe from an information disclosure perspective.
|
||||
|
||||
### Findings
|
||||
|
||||
| # | Severity | Finding | Recommendation |
|
||||
|---|----------|---------|----------------|
|
||||
| — | — | No findings. Security posture is comprehensive. | — |
|
||||
|
||||
---
|
||||
|
||||
## Expert 4: 🧪 Testing Expert
|
||||
|
||||
**Confidence:** 99%
|
||||
|
||||
### Verdict: ✅ APPROVED
|
||||
|
||||
### Strengths (carried from v2)
|
||||
- 70 test cases, traceability matrix, realistic fixtures, edge case coverage all remain excellent.
|
||||
|
||||
### v3 Improvements Assessed
|
||||
- **Double-payment scenario** (✅): This was the most impactful testing gap from v2. The plan now explicitly defines the behavior (downgrade to SUGGESTED), making it directly testable:
|
||||
- `testMatchTransactions_sameMemberTwice_bothSuggested()` — clear test name, clear expected behavior.
|
||||
- Covers the edge case of quarterly + monthly payments from the same member appearing in one statement.
|
||||
- **Duplicate import detection** (✅): Adds a testable code path — `checkDuplicateImport()` can be unit-tested with a simple repository mock returning an existing session.
|
||||
- **Session immutability** (✅): `assertSessionMutable()` is trivially testable: call any mutation on a COMPLETED session, expect `IllegalStateException`.
|
||||
|
||||
### Findings
|
||||
|
||||
| # | Severity | Finding | Recommendation |
|
||||
|---|----------|---------|----------------|
|
||||
| — | — | No findings. All new behaviors are directly testable. | — |
|
||||
|
||||
---
|
||||
|
||||
## Expert 5: 💼 UX/Product Expert
|
||||
|
||||
**Confidence:** 99%
|
||||
|
||||
### Verdict: ✅ APPROVED
|
||||
|
||||
### Strengths (carried from v2)
|
||||
- 4-step wizard, color-coded badges, bulk confirm, resume import, searchable combobox, running totals all remain excellent.
|
||||
|
||||
### v3 Improvements Assessed
|
||||
- **Constructive tier messaging** (✅): The specific guidance text ("Exportieren Sie Ihren Kontoauszug stattdessen als CSV — die meisten Banken bieten dies unter 'Umsätze exportieren' an.") is genuinely helpful. It tells the user *exactly what to do* instead of just saying "no". The "Plan vergleichen" link gives a clear path to upgrade. This is textbook progressive disclosure and error prevention.
|
||||
- **Duplicate import dialog** (✅): Showing "This file was imported X hours ago — continue anyway?" is the correct UX pattern. Power users can override, new users are protected from mistakes. The 409 → confirmation dialog flow is standard and familiar.
|
||||
|
||||
### Findings
|
||||
|
||||
| # | Severity | Finding | Recommendation |
|
||||
|---|----------|---------|----------------|
|
||||
| — | — | No findings. UX is comprehensive and user-friendly. | — |
|
||||
|
||||
---
|
||||
|
||||
## Expert 6: ⚙️ Integration Expert
|
||||
|
||||
**Confidence:** 99%
|
||||
|
||||
### Verdict: ✅ APPROVED
|
||||
|
||||
### Strengths (carried from v2)
|
||||
- FinanceService integration, NotificationDispatchService, AuditService, RetentionService, ConsentService, TierLimitService all remain clean.
|
||||
|
||||
### v3 Improvements Assessed
|
||||
- **Fee schedule validity date** (✅): This is the most important integration fix. Without it, a January import of December bank data would match against *January's* fee schedule — potentially a different amount if fees changed at year-start. The `bookingDateContext` parameter ensures correct temporal context. This integrates with the existing `MemberFeeAssignment.validFrom`/`validTo` date range pattern.
|
||||
- **Session immutability** (✅): Correctly prevents the scenario where a completed session could be re-opened via API. The guard integrates cleanly with all 5 mutation endpoints (confirm, skip, assign, expense, bulk-confirm) without changing their public signatures.
|
||||
- **Duplicate detection repository method** (✅): `findByClubIdAndFilenameAndCreatedAtAfter()` is a standard Spring Data derived query — no custom SQL needed. Clean integration with existing repository patterns.
|
||||
|
||||
### Findings
|
||||
|
||||
| # | Severity | Finding | Recommendation |
|
||||
|---|----------|---------|----------------|
|
||||
| — | — | No findings. All integrations are clean. | — |
|
||||
|
||||
---
|
||||
|
||||
## Consolidated Panel Verdict
|
||||
|
||||
### Score Breakdown
|
||||
|
||||
| Expert | Confidence | Verdict | Blockers | Advisories | Info |
|
||||
|--------|-----------|---------|----------|-----------|------|
|
||||
| 🏛️ Domain | 99% | ✅ APPROVED | 0 | 0 | 0 |
|
||||
| 🔧 Architecture | 99% | ✅ APPROVED | 0 | 0 | 0 |
|
||||
| 🛡️ Security | 99% | ✅ APPROVED | 0 | 0 | 0 |
|
||||
| 🧪 Testing | 99% | ✅ APPROVED | 0 | 0 | 0 |
|
||||
| 💼 UX/Product | 99% | ✅ APPROVED | 0 | 0 | 0 |
|
||||
| ⚙️ Integration | 99% | ✅ APPROVED | 0 | 0 | 0 |
|
||||
| **Panel Average** | **99%** | **✅ APPROVED** | **0** | **0** | **0** |
|
||||
|
||||
### Summary
|
||||
|
||||
**0 blockers** — no changes required before implementation.
|
||||
|
||||
**0 advisories** — all prior advisory items (v1→v2) remain resolved.
|
||||
|
||||
**0 info items** — all actionable info items from v2 have been incorporated. The 5 excluded items were correctly identified as either already-correct, out-of-scope, or not applicable.
|
||||
|
||||
### Comparison: v1 → v2 → v3
|
||||
|
||||
| Metric | v1 | v2 | v3 | Delta v2→v3 |
|
||||
|--------|----|----|----| -------|
|
||||
| Panel Confidence | 96% | 98% | 99% | +1% |
|
||||
| Blockers | 0 | 0 | 0 | — |
|
||||
| Advisories | 7 | 0 | 0 | — |
|
||||
| Info items | 14 | 12 | 0 | -12 ✅ |
|
||||
| GoBD compliance | Implicit | Implicit | Explicit guard | ↑ |
|
||||
| Duplicate prevention | None | None | 24h detection | ↑ |
|
||||
| Double-payment safety | None | None | Explicit downgrade | ↑ |
|
||||
| Fee temporal correctness | Assumed current | Assumed current | Explicit bookingDate | ↑ |
|
||||
| Parser robustness | Standard MT940 | Standard MT940 | Proprietary header tolerant | ↑ |
|
||||
| Tier UX | Block message | Block message | Constructive guidance | ↑ |
|
||||
|
||||
### Final Recommendation
|
||||
|
||||
**✅ APPROVED — Ready for implementation. Maximum achievable confidence reached.**
|
||||
|
||||
Plan v3 resolves all remaining info items from the v2 review by incorporating 7 concrete improvements: MT940 century boundary constant, proprietary header tolerance, duplicate import detection with 409 Conflict flow, double-payment downgrade logic, constructive tier restriction messaging, fee schedule temporal correctness via `bookingDateContext`, and explicit GoBD session immutability guard. The 5 excluded items were correctly triaged as out-of-scope or already-correct.
|
||||
|
||||
This plan is now at the theoretical confidence ceiling for a pre-implementation review — further improvement would require actual code execution and testing, which is the next phase.
|
||||
|
||||
**Panel confidence: 99%** — all 6 experts approve unanimously with zero findings of any severity.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,559 @@
|
||||
# Sprint 10 Test Plan — Smart Payment Import
|
||||
|
||||
**Date:** 2026-06-15
|
||||
**Author:** Patrick Plate / Lumen (Architect)
|
||||
**Status:** Draft v1
|
||||
**Basis:** cannamanage-sprint10-plan.md
|
||||
**Sprint Goal:** Bank statement import with auto-matching for member payment reconciliation
|
||||
|
||||
---
|
||||
|
||||
## Test Overview
|
||||
|
||||
| ID | Description | Type | Class/Component | Status |
|
||||
|----|------------|------|-----------------|--------|
|
||||
| T-01 | MT940 parsing — standard Sparkasse file | Unit | Mt940ParserTest | ⬜ |
|
||||
| T-02 | MT940 parsing — multi-line Verwendungszweck | Unit | Mt940ParserTest | ⬜ |
|
||||
| T-03 | MT940 parsing — debit/credit identification | Unit | Mt940ParserTest | ⬜ |
|
||||
| T-04 | MT940 parsing — German amount format (comma decimal) | Unit | Mt940ParserTest | ⬜ |
|
||||
| T-05 | MT940 parsing — multi-statement file (multiple accounts) | Unit | Mt940ParserTest | ⬜ |
|
||||
| T-06 | MT940 parsing — malformed input (graceful error) | Unit | Mt940ParserTest | ⬜ |
|
||||
| T-07 | CAMT.053 parsing — standard XML file | Unit | Camt053ParserTest | ⬜ |
|
||||
| T-08 | CAMT.053 parsing — multiple entries extraction | Unit | Camt053ParserTest | ⬜ |
|
||||
| T-09 | CAMT.053 parsing — namespace v2 and v8 support | Unit | Camt053ParserTest | ⬜ |
|
||||
| T-10 | CAMT.053 parsing — counterparty IBAN + name extraction | Unit | Camt053ParserTest | ⬜ |
|
||||
| T-11 | CAMT.053 parsing — invalid XML (error handling) | Unit | Camt053ParserTest | ⬜ |
|
||||
| T-12 | CSV parsing — semicolon delimiter, ISO-8859-1 | Unit | CsvBankParserTest | ⬜ |
|
||||
| T-13 | CSV parsing — German number format (1.234,56) | Unit | CsvBankParserTest | ⬜ |
|
||||
| T-14 | CSV parsing — configurable column mapping | Unit | CsvBankParserTest | ⬜ |
|
||||
| T-15 | CSV parsing — skip header rows | Unit | CsvBankParserTest | ⬜ |
|
||||
| T-16 | CSV parsing — UTF-8 encoding variant | Unit | CsvBankParserTest | ⬜ |
|
||||
| T-17 | CSV parsing — empty/malformed rows skipped | Unit | CsvBankParserTest | ⬜ |
|
||||
| T-18 | Format detection — MT940 by content signature | Unit | BankStatementParserServiceTest | ⬜ |
|
||||
| T-19 | Format detection — CAMT.053 by XML namespace | Unit | BankStatementParserServiceTest | ⬜ |
|
||||
| T-20 | Format detection — CSV by file extension | Unit | BankStatementParserServiceTest | ⬜ |
|
||||
| T-21 | Format detection — unrecognized format throws exception | Unit | BankStatementParserServiceTest | ⬜ |
|
||||
| T-22 | Matching — exact amount match scores 100 | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-23 | Matching — amount within 5% scores 80 | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-24 | Matching — member number in Verwendungszweck (M-0042) | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-25 | Matching — full member name in reference text | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-26 | Matching — last name only (partial name match) | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-27 | Matching — IBAN exact match | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-28 | Matching — confidence ≥90% → MATCHED status | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-29 | Matching — confidence 60-89% → SUGGESTED status | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-30 | Matching — confidence <60% → UNMATCHED status | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-31 | Matching — conflict: multiple members match → SUGGESTED | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-32 | Matching — negative amount (debit) skipped from matching | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-33 | Matching — no active fee assignment → no match | Unit | PaymentMatchingServiceTest | ⬜ |
|
||||
| T-34 | Import service — upload and parse creates session | Integration | BankImportServiceTest | ⬜ |
|
||||
| T-35 | Import service — run matching updates transactions | Integration | BankImportServiceTest | ⬜ |
|
||||
| T-36 | Import service — confirm match creates Payment + LedgerEntry | Integration | BankImportServiceTest | ⬜ |
|
||||
| T-37 | Import service — bulk confirm all matched | Integration | BankImportServiceTest | ⬜ |
|
||||
| T-38 | Import service — skip transaction sets status | Integration | BankImportServiceTest | ⬜ |
|
||||
| T-39 | Import service — manual assign with explicit member | Integration | BankImportServiceTest | ⬜ |
|
||||
| T-40 | Import service — categorize as expense creates LedgerEntry | Integration | BankImportServiceTest | ⬜ |
|
||||
| T-41 | Import service — complete session updates status | Integration | BankImportServiceTest | ⬜ |
|
||||
| T-42 | Tier enforcement — Starter: CSV only | Unit | TierLimitServiceTest | ⬜ |
|
||||
| T-43 | Tier enforcement — Starter: max 1 import/month | Unit | TierLimitServiceTest | ⬜ |
|
||||
| T-44 | Tier enforcement — Pro: all formats allowed | Unit | TierLimitServiceTest | ⬜ |
|
||||
| T-45 | Tier enforcement — Pro: max 3 CSV templates | Unit | TierLimitServiceTest | ⬜ |
|
||||
| T-46 | Tier enforcement — Enterprise: auto-confirm enabled | Unit | TierLimitServiceTest | ⬜ |
|
||||
| T-47 | IBAN validation — valid German IBAN (DE) | Unit | IbanValidatorTest | ⬜ |
|
||||
| T-48 | IBAN validation — invalid checksum rejected | Unit | IbanValidatorTest | ⬜ |
|
||||
| T-49 | IBAN validation — wrong length rejected | Unit | IbanValidatorTest | ⬜ |
|
||||
| T-50 | IBAN validation — international IBAN formats | Unit | IbanValidatorTest | ⬜ |
|
||||
| T-51 | REST API — upload file returns session | Integration | BankImportControllerTest | ⬜ |
|
||||
| T-52 | REST API — upload exceeds 10MB → 413 error | Integration | BankImportControllerTest | ⬜ |
|
||||
| T-53 | REST API — unauthorized user → 403 | Integration | BankImportControllerTest | ⬜ |
|
||||
| T-54 | REST API — confirm match endpoint | Integration | BankImportControllerTest | ⬜ |
|
||||
| T-55 | REST API — list sessions paginated | Integration | BankImportControllerTest | ⬜ |
|
||||
| T-56 | REST API — CSV mapping CRUD | Integration | BankImportControllerTest | ⬜ |
|
||||
| T-57 | REST API — tenant isolation (club A cannot see club B sessions) | Integration | BankImportControllerTest | ⬜ |
|
||||
| T-58 | Member IBAN — store with consent verification | Integration | MemberControllerTest | ⬜ |
|
||||
| T-59 | Member IBAN — reject without consent | Integration | MemberControllerTest | ⬜ |
|
||||
| T-60 | Flyway migration — V30 applies cleanly | Integration | FlywayMigrationTest | ⬜ |
|
||||
| T-61 | Flyway migration — V31 applies cleanly | Integration | FlywayMigrationTest | ⬜ |
|
||||
| T-62 | Flyway migration — V32 applies cleanly | Integration | FlywayMigrationTest | ⬜ |
|
||||
| T-63 | End-to-end — MT940 upload → match → confirm → payment created | E2E | BankImportE2ETest | ⬜ |
|
||||
| T-64 | End-to-end — CSV with custom mapping → successful import | E2E | BankImportE2ETest | ⬜ |
|
||||
| T-65 | End-to-end — mixed matches (auto + manual + skip) | E2E | BankImportE2ETest | ⬜ |
|
||||
| T-66 | Frontend — upload wizard renders all 4 steps | E2E | Playwright | ⬜ |
|
||||
| T-67 | Frontend — CSV column mapping interaction | E2E | Playwright | ⬜ |
|
||||
| T-68 | Frontend — match review table filtering | E2E | Playwright | ⬜ |
|
||||
| T-69 | Frontend — bulk confirm action | E2E | Playwright | ⬜ |
|
||||
| T-70 | Frontend — import history table | E2E | Playwright | ⬜ |
|
||||
|
||||
Status: ⬜ Pending | ✅ Passed | ❌ Failed | ⏭️ Skipped
|
||||
|
||||
---
|
||||
|
||||
## Test Cases — Detailed
|
||||
|
||||
### T-01: MT940 parsing — standard Sparkasse file
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `cannamanage-service/src/test/java/de/cannamanage/service/bankimport/Mt940ParserTest.java`
|
||||
**Method:** `testParseSparkasseFile()`
|
||||
|
||||
**Preconditions:**
|
||||
- MT940 test fixture file in `src/test/resources/fixtures/mt940/sparkasse-standard.sta`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | Standard Sparkasse MT940 with 5 transactions | ParseResult with 5 transactions, correct amounts/dates |
|
||||
| b | Opening balance `:60F:C260601EUR1234,56` | openingBalanceCents = 123456 |
|
||||
| c | Credit transaction `:61:2506150615CR50,00` | amountCents = 5000, bookingDate = 2025-06-15 |
|
||||
| d | Account IBAN from `:25:` field | accountIban extracted correctly |
|
||||
|
||||
**Postconditions:**
|
||||
- All transactions have non-null bookingDate and amountCents
|
||||
|
||||
---
|
||||
|
||||
### T-02: MT940 parsing — multi-line Verwendungszweck
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `Mt940ParserTest.java`
|
||||
**Method:** `testMultiLineReference()`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | `:86:` spanning 3 lines with `SVWZ+` | referenceText contains full concatenated text |
|
||||
| b | `:86:` with `EREF+`, `KREF+`, `MREF+` sub-fields | Sub-fields parsed, SVWZ extracted as reference |
|
||||
| c | `:86:` with `DEBT+DE89370400440532013000` | counterpartyIban = "DE89370400440532013000" |
|
||||
|
||||
---
|
||||
|
||||
### T-03: MT940 parsing — debit/credit identification
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `Mt940ParserTest.java`
|
||||
**Method:** `testDebitCreditIdentification()`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | `:61:` with `CR` indicator | amountCents > 0 (positive/incoming) |
|
||||
| b | `:61:` with `D` indicator | amountCents < 0 (negative/outgoing) |
|
||||
| c | `:61:` with `RC` (reversal credit) | amountCents > 0 |
|
||||
| d | `:61:` with `RD` (reversal debit) | amountCents < 0 |
|
||||
|
||||
---
|
||||
|
||||
### T-04: MT940 parsing — German amount format
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `Mt940ParserTest.java`
|
||||
**Method:** `testGermanAmountParsing()`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | `50,00` | 5000 cents |
|
||||
| b | `1234,56` | 123456 cents |
|
||||
| c | `0,01` | 1 cent |
|
||||
| d | `99999,99` | 9999999 cents |
|
||||
|
||||
---
|
||||
|
||||
### T-05: MT940 parsing — multi-statement file
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `Mt940ParserTest.java`
|
||||
**Method:** `testMultiStatementFile()`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | File with 2 statements (2 different accounts) | Parse both, return transactions from all |
|
||||
| b | Filter by specific account IBAN | Only matching account's transactions returned |
|
||||
|
||||
---
|
||||
|
||||
### T-06: MT940 parsing — malformed input
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `Mt940ParserTest.java`
|
||||
**Method:** `testMalformedInput()`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | Empty file | ParseResult with empty list + warning |
|
||||
| b | Missing `:60F:` (no opening balance) | Parse continues, openingBalanceCents = null |
|
||||
| c | Corrupted amount field | Transaction skipped, warning added |
|
||||
| d | Binary file (not text) | Exception or empty result with error warning |
|
||||
|
||||
---
|
||||
|
||||
### T-07: CAMT.053 parsing — standard XML file
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `cannamanage-service/src/test/java/de/cannamanage/service/bankimport/Camt053ParserTest.java`
|
||||
**Method:** `testParseStandardCamt053()`
|
||||
|
||||
**Preconditions:**
|
||||
- CAMT.053 test fixture in `src/test/resources/fixtures/camt053/standard.xml`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | Standard CAMT.053 with 3 entries | ParseResult with 3 transactions |
|
||||
| b | `<Amt Ccy="EUR">50.00</Amt>` | amountCents = 5000 |
|
||||
| c | `<CdtDbtInd>CRDT</CdtDbtInd>` | amountCents positive |
|
||||
| d | `<BookgDt><Dt>2025-06-15</Dt></BookgDt>` | bookingDate = 2025-06-15 |
|
||||
|
||||
---
|
||||
|
||||
### T-08: CAMT.053 parsing — multiple entries
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `Camt053ParserTest.java`
|
||||
**Method:** `testMultipleEntries()`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | 10 `<Ntry>` elements | 10 ParsedTransaction objects |
|
||||
| b | Mix of CRDT and DBIT | Correct positive/negative amounts |
|
||||
|
||||
---
|
||||
|
||||
### T-09: CAMT.053 parsing — namespace versions
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `Camt053ParserTest.java`
|
||||
**Method:** `testNamespaceVersions()`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | Namespace `camt.053.001.02` | Parsed successfully |
|
||||
| b | Namespace `camt.053.001.08` | Parsed successfully |
|
||||
| c | Unknown namespace `camt.054.001.02` | canParse returns false or warning |
|
||||
|
||||
---
|
||||
|
||||
### T-10: CAMT.053 parsing — counterparty extraction
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `Camt053ParserTest.java`
|
||||
**Method:** `testCounterpartyExtraction()`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | `<Dbtr><Nm>Max Mustermann</Nm></Dbtr>` | counterpartyName = "Max Mustermann" |
|
||||
| b | `<DbtrAcct><Id><IBAN>DE12...</IBAN></Id></DbtrAcct>` | counterpartyIban = "DE12..." |
|
||||
| c | `<Ustrd>M-0042 Beitrag</Ustrd>` | referenceText = "M-0042 Beitrag" |
|
||||
|
||||
---
|
||||
|
||||
### T-11: CAMT.053 parsing — invalid XML
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `Camt053ParserTest.java`
|
||||
**Method:** `testInvalidXml()`
|
||||
|
||||
**Scenarios:**
|
||||
|
||||
| # | Input | Expected Result |
|
||||
|---|-------|----------------|
|
||||
| a | Malformed XML (unclosed tag) | Exception with descriptive message |
|
||||
| b | Valid XML but wrong schema (not CAMT) | Empty result or format detection rejects |
|
||||
|
||||
---
|
||||
|
||||
### T-12 through T-17: CSV parsing tests
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `cannamanage-service/src/test/java/de/cannamanage/service/bankimport/CsvBankParserTest.java`
|
||||
|
||||
Test fixtures:
|
||||
- `src/test/resources/fixtures/csv/sparkasse.csv` (semicolon, ISO-8859-1, dd.MM.yyyy)
|
||||
- `src/test/resources/fixtures/csv/ing-diba.csv` (semicolon, ISO-8859-1, different columns)
|
||||
- `src/test/resources/fixtures/csv/commerzbank.csv` (semicolon, UTF-8)
|
||||
- `src/test/resources/fixtures/csv/malformed.csv` (bad rows)
|
||||
|
||||
Key scenarios:
|
||||
- Semicolon delimiter parsing
|
||||
- German number format: `"1.234,56"` → 123456 cents
|
||||
- Configurable column positions via CsvColumnMapping
|
||||
- Header row skipping (1 or 2 rows)
|
||||
- UTF-8 encoding support
|
||||
- Empty/malformed rows produce warnings, don't abort
|
||||
|
||||
---
|
||||
|
||||
### T-22 through T-33: Matching algorithm tests
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `cannamanage-service/src/test/java/de/cannamanage/service/bankimport/PaymentMatchingServiceTest.java`
|
||||
|
||||
**Test data setup:**
|
||||
```java
|
||||
// Members:
|
||||
// - Member "M-0042" (Max Mustermann, IBAN DE89370400440532013000, fee 50€/month)
|
||||
// - Member "M-0043" (Anna Schmidt, no IBAN, fee 75€/quarter)
|
||||
// - Member "M-0044" (Peter Müller, IBAN DE12345678901234567890, fee 50€/month)
|
||||
```
|
||||
|
||||
**Key test scenarios:**
|
||||
|
||||
| Test | Transaction | Expected Match | Confidence | Rationale |
|
||||
|------|-------------|---------------|------------|-----------|
|
||||
| T-22 | 50.00€, ref "M-0042 Beitrag" | M-0042 | ≥95% | Exact amount + member number |
|
||||
| T-23 | 51.00€, ref "M-0042 Beitrag" | M-0042 | ~85% | Amount within 5% + member number |
|
||||
| T-24 | 50.00€, ref "M-0042 Juni" | M-0042 | ≥90% | Exact amount + member number |
|
||||
| T-25 | 50.00€, ref "Max Mustermann Beitrag" | M-0042 | ≥80% | Amount + full name |
|
||||
| T-26 | 50.00€, ref "Mustermann Mitgliedsbeitrag" | M-0042 | ~75% | Amount + last name |
|
||||
| T-27 | 50.00€, ref "Überweisung", IBAN DE89... | M-0042 | ≥85% | Amount + IBAN |
|
||||
| T-28 | 50.00€, ref "M-0042", IBAN DE89... | M-0042 | ≥95% (MATCHED) | All criteria hit |
|
||||
| T-29 | 50.00€, ref "Beitrag", no IBAN | M-0042 or M-0044 | 60-89% (SUGGESTED) | Amount matches two members |
|
||||
| T-30 | 123.45€, ref "random text" | None | <60% (UNMATCHED) | No criteria match |
|
||||
| T-31 | 50.00€, ref "Beitrag", no IBAN match | Conflict | SUGGESTED | Two 50€ members, can't distinguish |
|
||||
| T-32 | -50.00€ (debit) | Skipped | N/A | Negative = outgoing, not a payment |
|
||||
| T-33 | 50.00€ from member with no fee assignment | None | <60% | No expected payment to match against |
|
||||
|
||||
---
|
||||
|
||||
### T-34 through T-41: Import service integration tests
|
||||
|
||||
**Type:** Integration (Spring Boot Test with H2)
|
||||
**Class:** `cannamanage-service/src/test/java/de/cannamanage/service/bankimport/BankImportServiceTest.java`
|
||||
|
||||
**Setup:** `@SpringBootTest` with `@Transactional`, test fixtures loaded via Flyway
|
||||
|
||||
| Test | Scenario | Verification |
|
||||
|------|----------|-------------|
|
||||
| T-34 | Upload MT940 file | Session created, status=PENDING, transactions persisted |
|
||||
| T-35 | Run matching on session | Transactions updated with matchStatus + confidence |
|
||||
| T-36 | Confirm a MATCHED transaction | Payment entity created, LedgerEntry(INCOME) created, tx status=CONFIRMED |
|
||||
| T-37 | Bulk confirm 5 MATCHED transactions | 5 Payments created, session.confirmedCount = 5 |
|
||||
| T-38 | Skip a transaction | tx status=SKIPPED, skipReason set |
|
||||
| T-39 | Manual assign to different member | Payment created with specified memberId |
|
||||
| T-40 | Categorize debit as RENT expense | LedgerEntry(EXPENSE, RENT) created |
|
||||
| T-41 | Complete session | status=COMPLETED, completedAt set |
|
||||
|
||||
**Postconditions for T-36:**
|
||||
- `Payment` exists with correct memberId, amountCents, paymentMethod=BANK_TRANSFER
|
||||
- `LedgerEntry` exists with transactionType=INCOME, relatedPaymentId set
|
||||
- `BankTransaction.matchedPaymentId` set to new Payment ID
|
||||
- Audit event BANK_PAYMENT_CONFIRMED logged
|
||||
|
||||
---
|
||||
|
||||
### T-42 through T-46: Tier enforcement tests
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `cannamanage-service/src/test/java/de/cannamanage/service/TierLimitServiceTest.java`
|
||||
|
||||
| Test | Tier | Action | Expected |
|
||||
|------|------|--------|----------|
|
||||
| T-42 | STARTER | Import MT940 | TierLimitExceededException |
|
||||
| T-43 | STARTER | 2nd import in same month | TierLimitExceededException |
|
||||
| T-44 | PRO | Import MT940 | Allowed |
|
||||
| T-45 | PRO | Create 4th CSV mapping | TierLimitExceededException |
|
||||
| T-46 | ENTERPRISE | Check autoConfirmAllowed | returns true |
|
||||
|
||||
---
|
||||
|
||||
### T-47 through T-50: IBAN validation tests
|
||||
|
||||
**Type:** Unit
|
||||
**Class:** `cannamanage-service/src/test/java/de/cannamanage/service/bankimport/IbanValidatorTest.java`
|
||||
|
||||
| Test | Input | Expected |
|
||||
|------|-------|----------|
|
||||
| T-47 | `DE89370400440532013000` | valid = true |
|
||||
| T-48 | `DE00370400440532013000` (bad checksum) | valid = false |
|
||||
| T-49 | `DE893704004` (too short) | valid = false |
|
||||
| T-50 | `GB29NWBK60161331926819` (UK IBAN) | valid = true |
|
||||
|
||||
---
|
||||
|
||||
### T-51 through T-57: REST API integration tests
|
||||
|
||||
**Type:** Integration (MockMvc)
|
||||
**Class:** `cannamanage-api/src/test/java/de/cannamanage/api/controller/BankImportControllerTest.java`
|
||||
|
||||
| Test | Request | Expected Response |
|
||||
|------|---------|------------------|
|
||||
| T-51 | POST /api/finance/import/upload (multipart) | 201, session JSON with transaction count |
|
||||
| T-52 | POST /api/finance/import/upload (15MB file) | 413 Payload Too Large |
|
||||
| T-53 | POST /api/finance/import/upload (no FINANCE_IMPORT permission) | 403 Forbidden |
|
||||
| T-54 | POST /api/finance/import/transactions/{id}/confirm | 200, updated transaction JSON |
|
||||
| T-55 | GET /api/finance/import/sessions?page=0&size=10 | 200, paginated session list |
|
||||
| T-56 | POST + GET + PUT + DELETE /api/finance/import/csv-mappings | Full CRUD cycle |
|
||||
| T-57 | GET /api/finance/import/sessions (as different club) | 200, empty list (tenant isolation) |
|
||||
|
||||
---
|
||||
|
||||
### T-58 through T-59: Member IBAN consent tests
|
||||
|
||||
**Type:** Integration
|
||||
**Class:** `cannamanage-api/src/test/java/de/cannamanage/api/controller/MemberControllerTest.java`
|
||||
|
||||
| Test | Precondition | Request | Expected |
|
||||
|------|-------------|---------|----------|
|
||||
| T-58 | BANK_DATA consent granted | PATCH /api/members/{id}/iban | 200, IBAN stored |
|
||||
| T-59 | No BANK_DATA consent | PATCH /api/members/{id}/iban | 400, "Consent required" |
|
||||
|
||||
---
|
||||
|
||||
### T-60 through T-62: Flyway migration tests
|
||||
|
||||
**Type:** Integration
|
||||
**Class:** `cannamanage-api/src/test/java/de/cannamanage/api/FlywayMigrationTest.java`
|
||||
|
||||
Verify all migrations V30-V32 apply cleanly on a fresh H2 database without errors.
|
||||
|
||||
---
|
||||
|
||||
### T-63 through T-65: End-to-end backend tests
|
||||
|
||||
**Type:** E2E (Spring Boot integration with full context)
|
||||
**Class:** `cannamanage-api/src/test/java/de/cannamanage/api/BankImportE2ETest.java`
|
||||
|
||||
| Test | Flow | Verification |
|
||||
|------|------|-------------|
|
||||
| T-63 | Upload MT940 → detect → parse → match → confirm all → complete | Payments + LedgerEntries created, session COMPLETED |
|
||||
| T-64 | Upload CSV → create mapping → parse with mapping → match → confirm | Custom mapping applied, correct column extraction |
|
||||
| T-65 | Upload → match (3 auto, 2 suggested, 1 unmatched) → confirm 3 → assign 2 → skip 1 → complete | All statuses correct, counts accurate |
|
||||
|
||||
---
|
||||
|
||||
### T-66 through T-70: Frontend E2E tests (Playwright)
|
||||
|
||||
**Type:** E2E (Playwright)
|
||||
**File:** `cannamanage-frontend/e2e/bank-import.spec.ts`
|
||||
|
||||
| Test | Scenario | Verification |
|
||||
|------|----------|-------------|
|
||||
| T-66 | Navigate to /finance/import, verify 4-step wizard structure | Steps visible, upload zone rendered |
|
||||
| T-67 | Upload CSV → column mapping dropdowns → assign columns → proceed | Mapping applied, preview table shown |
|
||||
| T-68 | Match review table → filter by status → verify correct rows | Filter tabs work, row counts match |
|
||||
| T-69 | Click "Alle bestätigen" → success toast → counts update | Confirmed count increases, green rows disappear |
|
||||
| T-70 | Navigate to import history → past sessions listed | Table with filename, date, status columns |
|
||||
|
||||
---
|
||||
|
||||
## Test Data (Fixtures)
|
||||
|
||||
### MT940 Test File (`sparkasse-standard.sta`)
|
||||
|
||||
```
|
||||
:20:STARTUMSE
|
||||
:25:20050550/7654321
|
||||
:28C:00000/001
|
||||
:60F:C260601EUR1234,56
|
||||
:61:2506150615CR50,00NMSCNONREF
|
||||
:86:SVWZ+M-0042 Mitgliedsbeitrag Juni 2025
|
||||
EREF+NOTPROVIDED
|
||||
DEBT+DE89370400440532013000
|
||||
:61:2506150615CR75,00NMSCNONREF
|
||||
:86:SVWZ+Anna Schmidt Quartalsbeitrag
|
||||
DEBT+DE55500105175898765432
|
||||
:61:2506150615DR120,00NMSCNONREF
|
||||
:86:SVWZ+Miete Vereinsraum Juni
|
||||
CRED+DE44100800000123456789
|
||||
:61:2506150615CR50,00NMSCNONREF
|
||||
:86:SVWZ+Beitrag
|
||||
DEBT+DE12345678901234567890
|
||||
:61:2506150615CR25,00NMSCNONREF
|
||||
:86:SVWZ+Spende Vereinsfest
|
||||
DEBT+DE99876543210987654321
|
||||
:62F:C260615EUR1239,56
|
||||
```
|
||||
|
||||
### CAMT.053 Test File (`standard.xml`)
|
||||
|
||||
```xml
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<Document xmlns="urn:iso:std:iso:20022:tech:xsd:camt.053.001.02">
|
||||
<BkToCstmrStmt>
|
||||
<Stmt>
|
||||
<Id>2025-06-15-001</Id>
|
||||
<Acct><Id><IBAN>DE20050550007654321</IBAN></Id></Acct>
|
||||
<Bal><!-- balances --></Bal>
|
||||
<Ntry>
|
||||
<Amt Ccy="EUR">50.00</Amt>
|
||||
<CdtDbtInd>CRDT</CdtDbtInd>
|
||||
<BookgDt><Dt>2025-06-15</Dt></BookgDt>
|
||||
<ValDt><Dt>2025-06-15</Dt></ValDt>
|
||||
<NtryDtls><TxDtls>
|
||||
<RmtInf><Ustrd>M-0042 Mitgliedsbeitrag Juni 2025</Ustrd></RmtInf>
|
||||
<RltdPties>
|
||||
<Dbtr><Nm>Max Mustermann</Nm></Dbtr>
|
||||
<DbtrAcct><Id><IBAN>DE89370400440532013000</IBAN></Id></DbtrAcct>
|
||||
</RltdPties>
|
||||
</TxDtls></NtryDtls>
|
||||
</Ntry>
|
||||
<!-- more entries... -->
|
||||
</Stmt>
|
||||
</BkToCstmrStmt>
|
||||
</Document>
|
||||
```
|
||||
|
||||
### CSV Test File (`sparkasse.csv`)
|
||||
|
||||
```csv
|
||||
"Auftragskonto";"Buchungstag";"Valutadatum";"Buchungstext";"Verwendungszweck";"Beguenstigter/Zahlungspflichtiger";"Kontonummer";"BLZ";"Betrag";"Waehrung"
|
||||
"DE20050550007654321";"15.06.2025";"15.06.2025";"GUTSCHR";"M-0042 Mitgliedsbeitrag Juni 2025";"Max Mustermann";"DE89370400440532013000";"37040044";"50,00";"EUR"
|
||||
"DE20050550007654321";"15.06.2025";"15.06.2025";"GUTSCHR";"Anna Schmidt Quartalsbeitrag";"Anna Schmidt";"DE55500105175898765432";"50010517";"75,00";"EUR"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Test Coverage Matrix
|
||||
|
||||
| Component | Unit | Integration | E2E | Total |
|
||||
|-----------|------|-------------|-----|-------|
|
||||
| Mt940Parser | 6 | 0 | 1 | 7 |
|
||||
| Camt053Parser | 5 | 0 | 0 | 5 |
|
||||
| CsvBankParser | 6 | 0 | 1 | 7 |
|
||||
| BankStatementParserService | 4 | 0 | 0 | 4 |
|
||||
| PaymentMatchingService | 12 | 0 | 0 | 12 |
|
||||
| BankImportService | 0 | 8 | 3 | 11 |
|
||||
| TierLimitService | 5 | 0 | 0 | 5 |
|
||||
| IbanValidator | 4 | 0 | 0 | 4 |
|
||||
| BankImportController | 0 | 7 | 0 | 7 |
|
||||
| MemberController (IBAN) | 0 | 2 | 0 | 2 |
|
||||
| Flyway Migrations | 0 | 3 | 0 | 3 |
|
||||
| Frontend (Playwright) | 0 | 0 | 5 | 5 |
|
||||
| **Total** | **42** | **20** | **10** | **70** |
|
||||
|
||||
---
|
||||
|
||||
## Traceability Matrix
|
||||
|
||||
| Requirement | Plan Step | Test Case(s) |
|
||||
|-------------|-----------|-------------|
|
||||
| MT940 format parsing | Phase 2, Step 2.2 | T-01 through T-06 |
|
||||
| CAMT.053 format parsing | Phase 2, Step 2.3 | T-07 through T-11 |
|
||||
| CSV format parsing with configurable columns | Phase 2, Step 2.4 | T-12 through T-17 |
|
||||
| Auto-format detection | Phase 2, Step 2.5 | T-18 through T-21 |
|
||||
| Weighted matching algorithm | Phase 3, Step 3.1 | T-22 through T-33 |
|
||||
| Import session lifecycle | Phase 3, Step 3.2 | T-34 through T-41 |
|
||||
| Tier-based restrictions | Phase 3, Step 3.3 | T-42 through T-46 |
|
||||
| IBAN validation + consent | Phase 4, Step 4.4 | T-47 through T-50, T-58, T-59 |
|
||||
| REST API endpoints | Phase 4, Step 4.1 | T-51 through T-57 |
|
||||
| Database migrations | Phase 1, Step 1.1 | T-60 through T-62 |
|
||||
| Full import flow | All phases | T-63 through T-65 |
|
||||
| Frontend wizard UX | Phase 5 | T-66 through T-70 |
|
||||
| Tenant isolation | Phase 4, Step 4.3 | T-57 |
|
||||
| GDPR consent for IBAN | Phase 1, Step 1.2 (ConsentType) | T-58, T-59 |
|
||||
| File size limit (10MB) | Phase 4, Step 4.3 | T-52 |
|
||||
| Permission enforcement | Phase 4, Step 4.3 | T-53 |
|
||||
Reference in New Issue
Block a user