Iceberg and Cloudflare R2

curral attaches Iceberg REST catalogs through DuckDB’s iceberg extension. Cloudflare R2 Data Catalog is tested end to end.

Cloudflare R2 Data Catalog

No S3 secret is needed: R2 vends storage credentials through the catalog.

extensions: [httpfs, avro, iceberg]

secrets:
  - name: r2_catalog
    type: iceberg
    params:
      TOKEN: ${R2_TOKEN}

databases:
  - name: lake
    path: ${R2_WAREHOUSE}             # the Data Catalog's "Warehouse name"
    schema: ${R2_SCHEMA:-default}     # Iceberg namespace for unqualified names
    cache_ttl: 30s
    options:
      TYPE: iceberg
      SECRET: r2_catalog
      ENDPOINT: ${R2_CATALOG_URI}     # the Data Catalog's "Catalog URI"

default: lake

With external access off (the default), allow just the bucket:

curral serve --catalog catalog.yaml ... --allowed-path s3://<bucket>/

Use an R2 token with the narrowest scope possible. With a read-only token, not even a permissive policy can write.

Other Iceberg REST catalogs

Use an s3 secret for storage and an iceberg secret for the catalog:

extensions: [httpfs, iceberg]
secrets:
  - name: lake_s3
    type: s3
    params: { KEY_ID: "${AWS_ACCESS_KEY_ID}", SECRET: "${AWS_SECRET_ACCESS_KEY}", REGION: "${AWS_REGION:-us-east-1}" }
  - name: lake_catalog
    type: iceberg
    params:
      CLIENT_ID: ${ICEBERG_CLIENT_ID}
      CLIENT_SECRET: ${ICEBERG_CLIENT_SECRET}
      OAUTH2_SERVER_URI: ${ICEBERG_OAUTH_URI}
databases:
  - name: lake
    path: ${ICEBERG_WAREHOUSE}
    options: { TYPE: iceberg, SECRET: lake_catalog, ENDPOINT: "${ICEBERG_ENDPOINT}" }

Metadata cache (cache_ttl)

On every request DuckDB asks the REST catalog for the table metadata, and that is not cached across transactions. On R2 this round trip costs 250–900 ms, nearly all of the query time.

cache_ttl puts a local caching proxy between DuckDB and the catalog:

• Only 200 responses to GET are cached, keyed with a hash of the Authorization header.
• Identical concurrent requests collapse into a single upstream call.
• Writes (POST/PUT/DELETE) pass straight through and empty the cache.
• If a response carries *expires-at-ms, the entry expires one minute before the credentials do. R2 does not report expiry, so keep the TTL short.

Trade-off: commits by other writers can take up to cache_ttl to show up.

R2, simple aggregation over HTTP	no cache	cache_ttl: 30s
p50, 1 client	273 ms	8 ms
throughput, 8 clients	3 req/s	338 req/s

Things to know

• information_schema, duckdb_tables() and similar catalog functions are blocked for every role: two of them running at once over an Iceberg catalog crash the process (a DuckDB bug). Use /v1/schema instead.
• Non-SELECT statements that read Iceberg tables (INSERT ... SELECT, CREATE TABLE AS) arrive at the policy with resolved: false.
• To run curral against R2 on a server, follow Deploy with Docker Compose.