DAP CLI Tool Imports Structure but No Data into Tables

AzJedi

Hello Community,

I'm currently working with the DAP CLI tool to import Canvas Data 2 into a database, but I'm running into an issue where the tables and column headers are created successfully but the table rows remain empty. This is after running the “initdb” command (dap initdb --namespace canvas --table TABLENAME).

I am using the Command prompt as an admin in Windows 10 and Python version 3.12.2. I have tried both PostgreSQL and MySQL and have the same issue. It even says that “Data has been successfully retrieved” with a ‘.tsv.gs’ file mentioned. But when I look at the table, it just has empty rows. This happens with any of the tables I try to import into the database.

Has anyone else experienced this issue, or can anyone offer guidance on what might be going wrong or what I should try next?

Any insights or suggestions would be greatly appreciated!

LeventeHunyadi

Officially, Instructure supports Windows 11 for DAP client library but not Windows 10. You need to switch to one of the supported operating systems.

At a more technical level, the issue seems to originate from an open-source library that DAP client library uses called tsv2py. This library is responsible for efficiently parsing text data (specifically, tab-separated format) into Python in-memory data structures, which are then sent to the database driver. (Text manipulation done directly in Python is generally slow, and can easily become the bottleneck, this library brings approx. 5x speed improvement.) The latest version of this library published to PyPI is built for Windows 10.0.19045 (i.e. 2022 update of Windows 10) but if you have a less recent version of Windows, the library may fail (and hence no data be inserted/updated). Recently, we have contributed a patch to target Windows 8.1 and later for this library, which would be included in a future release. Likewise, this library uses AVX2 CPU vector instructions (SIMD, or single instruction, multiple data) on Intel/AMD machines. If you have an old machine, which lacks these CPU instructions, you might run into issues. All modern hardware for Windows has these instructions.