Batch insert calendar events programmatically

Hi!

I am trying to figure out a way to insert 12000+ calendar events to users in our Canvas account. Doing it through the API renders quite a few requests... To minimise the requests, I'm wondering if there's a way to batch insert all these calendar events?

Find more posts tagged with

Accepted answers

All comments

raj_veeravalli

Philip,

Have you investigated threading or multiprocessing options in your programming language of choice?

I am easily able to run 30 simultaneous requests and I can go to 50 if I want, which roughly equates to around 4-7 minutes execution time for 12000 requests.

If you program in Python there are some great example from @bbennett2 ‌ in the developer blog

https://community.canvaslms.com/groups/canvas-developers/blog

James

@philip4

There is not a way to insert multiple events with a single API call. The intent of the throttling is to keep you from overloading the system for others. You can make multiple requests, but the limit on how many you can make at one time is low.

Instead of batching them, there is another way to do it that has been pretty reliable for me.

Here's what I've discovered about pushing the limits. I am using Node JS and the Bottleneck library. There may be something a similar capability for your programming language. Most of my requests are GET, which probably have a lower cost that POST, so the numbers I give will change but the principles will the same.

The x-rate-limit-remaining starts at 700. When it gets to 0, things start to fail / stop.

There is a penalty of 50 for each simultaneous request. If you make 14 concurrent requests that had 0 cost, you would hit the pre-flight penalty. Since the calls do have cost, it will be less than in a sustained environment -- you might be able to get by with 14 one time, but the 15th would likely cause issues.

In the Bottleneck JS library, this is controlled by maxConcurrent and it is normally not the limiting factor for me. I can allow 40 or maybe 50 concurrent calls depending on the call -- but it doesn't ever hit that.

The limiting issue is making them all at the same time and with that pre-flight penalty, you can't get nearly that much.

The trick is to stagger the requests so that they don't all start at the same time.

Bottleneck has a minTime parameter, which specifies the number of milliseconds to wait between requests. The first one is made right away, but after that, there is a delay between requests to space them out.

I just got through revising my Obtaining and using Access Report data for an entire course script, which is entirely GET requests. I ended up with a minTime = 30 and maxConcurrent = 25.

The other thing I do with GETs is to limit the per_page to reduce the cost of the page, but that is not an issue with POST or PUT requests. Instead of using per_page=100 (the maximum allowed), I use per_page=50 or per_page=20 for expensive calls.

I could have pushed it more, but I was trying to make sure that it didn't time out with large classes so I sided with caution. With the current setup, my x-rate-limit-remaining is normally above 500.

It turns out that it is endpoint specific. There are three endpoints I use.

I fetch the sections for a course with a per_page limit of 50. The per_page limit of 50 is probably higher than it needs to be. At least at my institution, we never have more than a few sections in a page. This call isn't particularly expensive anyway. It's throttled, but it's not the bottleneck.
I fetch the list of users with their enrollments with a per_page limit of 20. This is an expensive call when the system is busy, which is why I went down to 20 per page. When I was doing 100 per page, I was sometimes seeing the x-rate-limit-remaining drop to below 200 with the second request (before using Bottleneck JS). Since this is an expensive call that takes a while to process, and limiting the number of simultaneous requests comes into play.
I also fetch the access report for every enrollment in the course. This is a non-API call, but it uses the x-rate-limit-remaining header as well. This is a fairly quick request, even with per_page=100, so I let it fetch 100 at a time. I could make a lot of these requests simultaneously, except for that pre-flight penalty. Even if I could run 50 of those at a time, and I probably could, I couldn't get up to 50 because of the pre-flight penalty. The limiting factor on this call is the minimum time between requests.

I start fetching my access reports as soon as I have the first 20 students. Then it gets the rest of the students before it finishes fetching the rest of the access reports.

I could set up two limiters, one for the enrollments, and one for the access reports. In other programs, I've waited for one type of request to finish before starting the second and change the parameters between types. This one has a front-end that people see in the browser, so I didn't want them to wait even longer but I still may make the change.

Anyway, for the getting of the enrollments, since it takes a long time to run, I want to cut back on the number that are running at the same time, so the minTime=30 and maxConcurrent=25 is really for this one. I ran into a few issues with minTime=30 and maxConcurrent=40. With a minTime=30, then you can only make 1000/30 = 33.33 requests per second, so setting maxConcurrent=40 was never getting hit when the requests are short, but these requests were taking more than 1 s to complete, so I was hitting the maxConcurrent limit and occasionally getting limiting errors.

For the access report calls, these were quick and not very expensive. I could go minTime=20 and maxConcurrent=50 and probably get by with it in a sustained setting -- provided my x-rate-limit-remaining had not already been drained by the enrollments. In my other program, I wait for one type to completely finish before starting the next time of fetch, so my x-rate-limit-remaining is almost back to 700 for each new type of request.

I had originally monitored the x-rate-limit-remaining throughout the process with my other program that downloads data nightly, but the Bottleneck JS library doesn't recognize changes made to existing calls in the queue, so the would timeout before the change I made took place. I just slowed the whole system down enough that it exhaust the x-rate-limit-remaining in the first place. That program averages about 10-11 calls per seconds, but it never times out since I slowed it down that much. It now takes 12 minutes to run instead of 9. Last night's run was only 3 minutes and 17 seconds, but we're between semesters and summer classes are fewer than fall or spring, so it's not a good indicator.

If you need to make 12000 requests, at 10 per second, you're only talking 1200 seconds or 20 minutes. Even at 5 calls per second, it's only 40 minutes. That's nothing to a server running a backend process.

You don't have to get these done instantly. Pushing the limits can make the system slower for others and getting events added to a calendar is probably less important than the user experience. You can make more than one request at a time, but you'll need to space them out to avoid that pre-flight penalty. If the calls take a while to complete or are really expensive, you may need to worry about maxConcurrent or you can just space them out more to get the same effect.

Unanswered Questions

Messaging Changes--- Suggestions for more streamlining
Hello, I was thinking of this a while ago and have been trying to message Canvas support about this but they suggested here but not which category. I apologize if this is the wrong spot. When messaging students right now we need to message one by one. I would like to know if there is a way we can change that to streamline…
Finding the LTIv1.3 IDs for Members of a Course
I'm building a tool for my institution that imports grades from another LTI tool into Canvas so instructors can automate late assignment policies not otherwise supported by Canvas. I'm trying to use the LTIv1.3 ID which is readily available on the LTI-side as a link to the student record on Canvas, but I can't find the ID…
Canvas Media Question
How can one track Canvas Media use without having the reports available from Canvas Studio? Checking API endpoints and the Data 2 schema, there do not seem to be dedicated Canvas Media endpoints. Am I missing some? A method I've considered is using the body field of the wiki_pages table, as this contains the html code of…
Canvas Catalog API - Completed Certificates
Hello, I'm trying to utilize an API call to pull completed certificates and was hoping to download and store the certs as a backup. I can successfully run the API call but it will not allow me to view the certificate unless I access the URL while signed into an account where it's "my" cert. Is there a permissions setting…
UI for weekly progression idea
UI for weekly progression idea that rest in the Schedule tab of the **K-12 UI** of Canvas it's incomplete and I really would not take it and push it into instance but I am going to post it here because in the past people like @James && @robotcars || @Steve_25 have had terrific input in the past... maybe w/their eyes and…