initdb for big tables aborts part way through

Question

While running the DAP 2.2 CLI on a Windows 11 machine, using Python 3.3.12, to push data into an Aurora PosgtresQL database, the data load aborts mid way through — it will look something like this:

For some of the medium-sized tables I truncated the table, re-tried and it eventually loaded all the data; for this biggest table, though, it has never gotten past 40 percent. Since there's a lot of data, I'd love to resume the process rather than restarting it.

To be clear, by abort what I mean is that init db stops doing anything I can observe — no error messages, no local CPU activity, no networking activity, no more data rows going into the database, even after waiting overnight. Eventually I kill the process.

Any suggestions on what might get the data loaded?

StevenWalls · Answer

We've had this exact issue ever since CD2 came out. Support had no resolution at first, then a couple of DAP versions ago, they changed their listed requirements to Windows 11 (via WSL-based Python only) which is not feasible for us.

DanGioia · Answer

The dap client frequently has issues on Windows. I had it timeout alot. If you do the same operation in a WSL environment on the same machine, in my experience, it works flawlessly

AndreDhondt · Answer

The files don't look corrupt to me (visual inspection of the .tsv files in Excel isn't foolproof since there are line feeds in the data) but it renders pretty well for most lines. The process doesn't abort at the same point every time either. I've seen it load zero rows (most of the time), ~107k rows, and ~220k rows — but it never got beyond 40% loaded.