Pull large body of submission scores via API

Question

Hey all!

I have a reporting use case to pull a large body of submission scores and am tentatively using the REST API to achieve this. Does anyone care to weigh in on the overall procedure I'm using and offer any feed back on this strategy, or alternatives to achieve the same end:

For each active term

→ Filter for courses matching a particular set of SIS IDs (roughly 90 courses right now)

→ Filter those courses for assignments in a set of assignment groups

→ Retrieve all submissions for those assignments

The total returned submissions can reach into the many tens of thousands in this case and this takes hours (or a day or two) to retrieve by http requests. Is there a better way to filter down to the relevant details? Would this be a good use case for either GraphQL or the Data Service? I'm not very familiar with those two items and would appreciate ruling them out early if they are not good candidates for these requirements.

Thanks!

melodyc_lam · Accepted Answer

@matt2 I'll let others chime in regarding graphQL/Canvas Data 2, but here are some opportunities to make things faster through the HTTPs route:

* Utilize the SIS Export reports to get the data for the courses that you need by active terms. These will be CSVs which then can get filtered.
* If you know which SIS IDs and which assignment groups you need, you can use a CSV file to record the IDs of the target assignment groups in the course (assuming they're named consistently across all courses), then utilize the Get Assignments by Assignment Group API call to get the assignments. As you add courses, just add rows containing the SIS IDs and the assignment groups to the CSV, that way you're not having to do multiple calls to the course to get their assignment groups then get assignments.
* Since you have SIS IDs you can use the special call to call into the API using the SIS ID as an identifier, instead of having to find the course ID.
* You may want to look at List Submissions for Multiple assignments as a way to get all the assignment submissions for multiple assignments, grouped by student.

If you can describe the use case a little more, for example, are the assignment groups the same name, are more assignments going to be placed in the groups over the term, are students the same in all of these courses, etc. we can help suggest more ways to make it faster. Hope this helps!

jerry_nguyen · Accepted Answer

@matt2 APIs are mainly used when you need to pull real-time data. Based on your scenario, it seems you don’t require up-to-the-minute data but rather data for analytical purposes. I would suggest using Canvas Data 2, as the data is typically only 2–4 hours behind the live system. Canvas Data allows you to run custom SQL queries, so you can apply filters based on your specific needs

Canvas Data 2 is not too difficult to set up. Canvas already provides most of the necessary tools to download the data and import it into a database. If you don’t yet have access to Canvas Data 2, you can contact your CSM to have it enabled for your Canvas instance. Feel free to PM me if you need help setting it up