Issues with Canvas Data 2 API Files download

Question

I’m working with Canvas Data 2 API and am having trouble accessing S3 files programmatically. While I can manually open the provided URLs in my browser, I receive a 403 Forbidden error when trying to access them using Python.

I’ve been provided with an S3 link to a file, but I can’t retrieve the file using Python. Here is a typical URL I’m working with:

https://data-access-platform-output-prod-iad.s3.amazonaws.com/output/...

When trying to access this URL using Python with requests or urllib, I consistently receive a 403 Forbidden error. However, I can access this URL successfully in the Brave browser.

Attempts:

* Basic HTTP GET Request
 
import requests
url = 'https://data-access-platform-output-prod-iad.s3.amazonaws.com/...'
response = requests.get(url)
if response.status_code == 200:
    with open('output_file.json.gz', 'wb') as f:
        f.write(response.content)
else:
    print(f"Failed to retrieve file: {response.status_code}")
* Including Headers
canvas_req_headers = {
'User-Agent': 'Mozilla/5.0 ...',
'Referer': 'https://data-access-platform-output-prod-iad.s3.amazonaws.com/...'
}
response = requests.get(url, headers=canvas_req_headers)
Can anyone provide guidance on how to access these files programmatically? Are there specific headers, authentication methods, or additional steps required for programmatic access that might not be needed for manual access?

Gergely Schmidt · Answer

Hi,

Have you you considered giving a try to our DAP CLI client? It does exactly what you are trying to do, and it works out-of-the box.
Here is the documentation and you can download it from pypi using pip:

pip install instructure-dap-client

Here is a beginners guide as well:

https://data-access-platform-api.s3.amazonaws.com/client/BEGINNERS-GUIDE.html