Hi everyone,
I'm exploring some changes to how the DAP Query API handles schemas, and I want to hear from you before we go too far down any path.
Here's the situation. The DAP schema today is fairly sophisticated. It defines specific enum values, type constraints, and so on. That gives you a strong data contract, but it has side effects you've probably noticed:
- When Canvas by Instructure adds a new enum value that isn't in the schema yet, we substitute
__dap_unspecified__ so we don't break your pipelines. You see the row, but the actual value is gone. - We hold back new columns and tables until they're fully modeled in the schema. The recent CD2 update is a good example of this pattern.
- We filter out records that don't fit the schema to protect the contract, which means you sometimes never see rows that exist upstream.
The alternative we're exploring is a simpler schema based on basic data types: strings, numbers, booleans, and timestamps. You'd get the raw values as Canvas produces them, including new enums on day one. The trade-off is that you'd own more of the validation on your side, and the contract would loosen.
Before we commit either way, I want to understand how the current setup serves you in practice:
- Do you actively use the strict schema (enum values, type constraints) for downstream validation, or is it mostly documentation for you?
- Have
__dap_unspecified__ values caused real pain, or are they invisible because the pipeline keeps running? - If we shipped raw types and published the enum catalogs separately, would that work for your stack?
- What would break for you if we went this direction?
Honest reactions welcome, including "please don't change anything." Thanks in advance.
