API: filter resources on timestamp or previous-sync cursor (Users, Courses, Sections & Enrollments)

We are using the Canvas REST API to synchronize users into our application.
Every 6 hours we are requesting:

- all users
- all coursers
- all section
- all enrollments

for 129 different schools/clients.

We then determine ourselves what was changed update those changes to our database.
While not ideal, this works.

We are however running in to problems with a new client.

This client has a huge amount of users (up to 70.000 per course) and lots of courses.
And wants to have (near) realtime updates, our current implementation is not sufficient to allow for this.
Even if we would just continuously synchronize all data, the time required to fetch and parse all that data is going to take to long.

We are looking at live events but as this is focussed towards analytics it fires massive amounts of (for us) un-usable events (especially user update events), -also.. we would like to avoid stream-processing if possible.

Ideally we would love to be able to add a filter to the existing endpoints:

last_update > timestamp

Is there any way where we can get to near realtime synchronization without causing excessive amounts of traffic?

*edit: typo

Find more posts tagged with

Accepted answers

All comments

jdh132

I happened to see your message in my recent daily digest this morning. You might want to look at Canvas Data 2 and is closer to what you want.
https://community.canvaslms.com/t5/Admin-Guide/What-is-Canvas-Data-2/ta-p/560956

This does allow you to pull individual table deltas on a regular schedule. I'm not familiar with what your current use case or application is but assuming an LTI approach wouldn't work either/

sendres

As others have said, Canvas Data 2 is the best option to get this much bulk data out of Canvas, but it comes with the ~ 4 hour latency.

I'm wonder if, rather than pulling this data out of Canvas as the source of truth, the institution could directly transfer its SIS files to you right after its import to Canvas concludes? If they're willing to cooperate, this would seem a straightforward way to get the data almost immediately.

sendres

Hi @FlipVernooij, if Canvas is the only source of truth for your project, then I reluctantly agree that you need the most efficient—or, more precisely, the least inefficient—way to get the data out.

Other than CD2 with its ~ 4 hour latency, the only other idea I have is to use the Account Reports API endpoint to generate a set of SIS export files. This gives you a set of CSV files equivalent to those generated manually by going to your Root Account > Settings > Reports > SIS Export. The process takes a while but might get you what you need.

I suggest manually running a set of SIS Export files; it's pretty self explanatory but you can see the Admin Guide documentation for more information. If those exports provide the information you're looking for, you can then automate the process.

As an example, here's a PowerShell script I use for this. The script does an API call to request the export, then invokes a callback function to wait for them to be completed before downloading the results. Beware: I never thought I'd be sharing this, so it might be a little rough. It's written in old PowerShell 5.1 and there may be some hard-coded variables you'll need to change. On the plus side, it's been reliable and has decent logging in case anything doesn't go to plan.

From what I can tell, the actual SIS report generation is fast, but your request will wait in a queue for a while. The logs from my most recent run show completion sat at 0% complete for 14 minutes, then jumped to 100% done within the next minute. Therefore, in theory, you could run a set of reports, wait for them to be created, import those results, and then immediately repeat the process. In practice, I don't know whether Instructure might balk at you running reports so many times a day. You could experiment with the scheduling to see what works in the real world.

Let me know if this does anything for you.

Unanswered Questions

UI for weekly progression idea
UI for weekly progression idea that rest in the Schedule tab of the **K-12 UI** of Canvas it's incomplete and I really would not take it and push it into instance but I am going to post it here because in the past people like @James && @robotcars || @Steve_25 have had terrific input in the past... maybe w/their eyes and…
add `desc` or `asc` to a few api calls that are already sorted by date\time\index such as `due_at`
A case for this argument: I live in thee K-12 realm which means we have way too many assignments... which means that at times a simple last 10 assignments of a course question(last 10 date wise) will take us 300 assignments to answer Solution for this example in particular: I would love to make this modified version of an…
Integrate a React App (Not Canvas) to be Canvas LMS SSO.
Hey Community: I want to ask first if Canvas LMS as a Identity provider is only supported with SAML? What other protocols are supported? My use case is that I have a React App hosted on cloud provider (AWS) and SpringBoot as a backend (Service provider). I want to integrate Canvas SSO as a Identity provider (IdP). I…
Donot have valid data to test certain APIs
Hi , I’m working on validating the API test cases for CANVAS Api and need suitable test data to verify the 200 OK status code responses. I am using free for teachers version account. PFA Ecel sheet about the end points I am checking for.
Getting Canvas LTI Data
I am attempting to programmatically access the LTI usage data presented in the Canvas Apps Dashboard (total launches, unique users, subaccounts, courses over time). Could anyone advise on the best method to replicate this information by using the Canvas Rest API calls or Canvas Data 2?