description: Data Lakehouse at-least-once delivery profile for AidboxTopicDestination. Default writeMode 'managed-zerobus' streams writes directly into a Databricks Unity Catalog managed Delta table via the Zerobus gRPC streaming-ingest SDK (no SQL warehouse on the hot path; pay-per-row ingest). Initial bulk export uses a temporary staging external Delta table on the customer's bucket and merges via the SQL warehouse on the resource id. 'managed-sql' targets the same managed table but routes every batch through the Databricks SQL warehouse (Statement Execution API) — same staging+MERGE for initial bulk; use this when Zerobus isn't available on the customer's SKU. 'external-direct' falls back to writing Parquet directly to an external Delta table on S3/GCS/ADLS via the Delta Kernel writer (no SQL warehouse, restart-safe-idempotent via Delta txn action). package_name: io.health-samurai.core.r4b derivation: constraint name: DataLakehouseAtLeastOnceProfile type: AidboxTopicDestination elements: kind: pattern: {type: String, value: data-lakehouse-at-least-once} index: 0 parameter: index: 1 slicing: rules: closed discriminator: - {path: name, type: pattern} min: null slices: databricksWorkspaceId: match: {name: databricksWorkspaceId} schema: short: Numeric Databricks workspace ID (e.g. 1234567890123456). Required when writeMode=managed-zerobus; used together with databricksRegion to compose the Zerobus gRPC server endpoint host. index: 114 elements: name: pattern: {type: String, value: databricksWorkspaceId} index: 115 value: choices: [valueString] index: 117 valueString: {type: string, choiceOf: value, index: 118} resource: {index: 119} part: {index: 120} required: [valueString, value] awsAccessKeyId: match: {name: awsAccessKeyId} schema: short: 'AWS access key ID (omit to use default provider chain: env / EC2 / EKS IRSA). Supports vault refs.' index: 23 elements: name: pattern: {type: String, value: awsAccessKeyId} index: 24 value: choices: [valueString] index: 26 valueString: {type: string, choiceOf: value, index: 27} resource: {index: 28} part: {index: 29} required: [valueString, value] sendIntervalMs: match: {name: sendIntervalMs} schema: short: Maximum time between commits in milliseconds _required: true index: 51 elements: name: pattern: {type: String, value: sendIntervalMs} index: 52 value: choices: [valueUnsignedInt] index: 54 valueUnsignedInt: {type: unsignedInt, choiceOf: value, index: 55} resource: {index: 56} part: {index: 57} required: [value, valueUnsignedInt] awsRegion: match: {name: awsRegion} schema: short: AWS region for the S3 bucket (required for real AWS; MinIO/Garage accept us-east-1) index: 16 elements: name: pattern: {type: String, value: awsRegion} index: 17 value: choices: [valueString] index: 19 valueString: {type: string, choiceOf: value, index: 20} resource: {index: 21} part: {index: 22} required: [valueString, value] databricksWarehouseId: match: {name: databricksWarehouseId} schema: short: Databricks SQL warehouse ID. Required when writeMode=managed-sql; the warehouse executes INSERT / COPY INTO / ALTER on behalf of this destination. Also required for managed-zerobus when initial-export is enabled (the staging-table schema sync still uses INFORMATION_SCHEMA via the warehouse). index: 100 elements: name: pattern: {type: String, value: databricksWarehouseId} index: 101 value: choices: [valueString] index: 103 valueString: {type: string, choiceOf: value, index: 104} resource: {index: 105} part: {index: 106} required: [valueString, value] stagingTablePath: match: {name: stagingTablePath} schema: short: S3/GCS/ADLS URI for an external Delta table used as staging during managed-sql or managed-zerobus mode's initial-export phase. Required when writeMode is managed-sql or managed-zerobus AND skipInitialExport is not true. The customer must pre-configure a UC external location at this path. index: 107 elements: name: pattern: {type: String, value: stagingTablePath} index: 108 value: choices: [valueString] index: 110 valueString: {type: string, choiceOf: value, index: 111} resource: {index: 112} part: {index: 113} required: [valueString, value] tableName: match: {name: tableName} schema: short: 'Unity Catalog table full name: catalog.schema.table. Required when databricksWorkspaceUrl is set.' index: 86 elements: name: pattern: {type: String, value: tableName} index: 87 value: choices: [valueString] index: 89 valueString: {type: string, choiceOf: value, index: 90} resource: {index: 91} part: {index: 92} required: [valueString, value] tablePath: match: {name: tablePath} schema: short: Delta table URI, e.g. s3://my-bucket/path/to/table. Must exist (created via Databricks UI / SQL). Required when databricksWorkspaceUrl is absent; ignored when set (table location resolved from Unity Catalog instead). index: 2 elements: name: pattern: {type: String, value: tablePath} index: 3 value: choices: [valueString] index: 5 valueString: {type: string, choiceOf: value, index: 6} resource: {index: 7} part: {index: 8} required: [valueString, value] viewDefinition: match: {name: viewDefinition} schema: short: Name of the ViewDefinition resource for data transformation _required: true index: 9 elements: name: pattern: {type: String, value: viewDefinition} index: 10 value: choices: [valueString] index: 12 valueString: {type: string, choiceOf: value, index: 13} resource: {index: 14} part: {index: 15} required: [valueString, value] s3Endpoint: match: {name: s3Endpoint} schema: short: Custom S3 endpoint (MinIO / Garage / LocalStack). Forces path-style URLs. index: 37 elements: name: pattern: {type: String, value: s3Endpoint} index: 38 value: choices: [valueString] index: 40 valueString: {type: string, choiceOf: value, index: 41} resource: {index: 42} part: {index: 43} required: [valueString, value] skipInitialExport: match: {name: skipInitialExport} schema: short: 'Skip initial export of existing data (default: false)' index: 72 elements: name: pattern: {type: String, value: skipInitialExport} index: 73 value: choices: [valueBoolean] index: 75 valueBoolean: {type: boolean, choiceOf: value, index: 76} resource: {index: 77} part: {index: 78} required: [value, valueBoolean] initialExportChunkCount: match: {name: initialExportChunkCount} schema: short: Number of hash-partitioned chunks the initial export splits into. Default 1 (single cursor). Chunks distribute across Aidbox pods and write into per-chunk staging Delta tables that are merged into the managed target in one MERGE INTO at finalize. index: 65 elements: name: pattern: {type: String, value: initialExportChunkCount} index: 66 value: choices: [valueUnsignedInt] index: 68 valueUnsignedInt: {type: unsignedInt, choiceOf: value, index: 69} resource: {index: 70} part: {index: 71} required: [value, valueUnsignedInt] writeMode: match: {name: writeMode} schema: short: Write path strategy. Default managed-zerobus (gRPC streaming-ingest direct into a Unity Catalog managed Delta table; no warehouse hours). Set managed-sql to route inserts through a Databricks SQL warehouse (same target, requires warm warehouse). Set external-direct to write directly into a non-managed external Delta table (S3 / MinIO / GCS / ADLS) via Kernel. index: 93 elements: name: pattern: {type: String, value: writeMode} index: 94 value: choices: [valueString] index: 96 valueString: {type: string, choiceOf: value, index: 97} resource: {index: 98} part: {index: 99} required: [valueString, value] batchSize: match: {name: batchSize} schema: short: Number of messages per Delta commit batch _required: true index: 44 elements: name: pattern: {type: String, value: batchSize} index: 45 value: choices: [valueUnsignedInt] index: 47 valueUnsignedInt: {type: unsignedInt, choiceOf: value, index: 48} resource: {index: 49} part: {index: 50} required: [value, valueUnsignedInt] databricksRegion: match: {name: databricksRegion} schema: short: AWS region of the Databricks workspace, e.g. us-east-1. Required when writeMode=managed-zerobus; used together with databricksWorkspaceId to compose the Zerobus gRPC server endpoint host. index: 121 elements: name: pattern: {type: String, value: databricksRegion} index: 122 value: choices: [valueString] index: 124 valueString: {type: string, choiceOf: value, index: 125} resource: {index: 126} part: {index: 127} required: [valueString, value] databricksWorkspaceUrl: match: {name: databricksWorkspaceUrl} schema: short: Databricks workspace URL, e.g. https://workspace.cloud.databricks.com. PRESENCE of this parameter activates Unity Catalog credential vending — tablePath and AWS credentials are then resolved from UC. index: 79 elements: name: pattern: {type: String, value: databricksWorkspaceUrl} index: 80 value: choices: [valueString] index: 82 valueString: {type: string, choiceOf: value, index: 83} resource: {index: 84} part: {index: 85} required: [valueString, value] targetFileSizeMb: match: {name: targetFileSizeMb} schema: short: Target Parquet file size in MiB during initial export (default 128). Avoid tiny files. index: 58 elements: name: pattern: {type: String, value: targetFileSizeMb} index: 59 value: choices: [valueUnsignedInt] index: 61 valueUnsignedInt: {type: unsignedInt, choiceOf: value, index: 62} resource: {index: 63} part: {index: 64} required: [value, valueUnsignedInt] awsSecretAccessKey: match: {name: awsSecretAccessKey} schema: short: AWS secret access key. Pair with awsAccessKeyId. Supports vault refs. index: 30 elements: name: pattern: {type: String, value: awsSecretAccessKey} index: 31 value: choices: [valueString] index: 33 valueString: {type: string, choiceOf: value, index: 34} resource: {index: 35} part: {index: 36} required: [valueString, value] package_version: 0.2605.0 class: profile kind: resource url: http://health-samurai.io/fhir/core/StructureDefinition/aidboxtopicdestination-dataLakehouseAtLeastOnceProfile base: http://health-samurai.io/fhir/core/StructureDefinition/AidboxTopicDestination version: 0.2605.0