Filtering noise from web_logs table for student activity

Our organization has been asked to provide data and reporting on student activity in Canvas, essentially the number of times a student has accessed courses and resources. The activity is stored in the "web_logs" table which records granular actions, similar to api calls and http requests. There is an abundance of noise in these logs and it is not documented how these logs can be parsed. There are several other Canvas data users that have posted on forums over the years trying to make sense of these logs, but I've seen many more questions than answers on this topic.

Has anyone in this community had success in extracting meaningful data from the web_logs table in Canvas? I know this is a long-shot, but please let me know if you have found an approach to working with reporting on student activity in Canvas. Thank you!

Find more posts tagged with

Canvas Data

Canvas Data 2

web logs

Accepted answers

All comments

Jeff_F

@MattG32857073 - I tend to filter out all of the lines that begin with /api. I also tend to look at the data in an aggregate view. That is, I don't try to get into very specific questions but much rather compile the data and analyze it for trends, I tend to distrust the completeness of log files to begin with but perhaps things have improved here in CD2. The documentation states: 'Logs include all interactions made with your instance of Canvas…'

To help answer your question, may I ask you provide more specifics to help focus in on what data you are looking to extract? For example, we might ask on average, how many times each week do instructors access Canvas Grades or SpeedGrader? Or how often do students view the Grades view? With that I can help with filter ideas.

Best, Jeff

MattG32857073

Thank you for your response, Jeff.

We are also looking at the web log data with the intention of analyzing basic trends and student adoption. It's less important for us to know what type of info they are accessing via Campus, but the number of Page Views and Participations would get us most of what we want to know. If it were easy, breaking the student activity down by which views were accessed (such as Grades) would be helpful, but non-essential. We're basically trying to recreate the Page View counts that we see in the Course Analytics page.

Here is the criteria I've been using, which is pretty specific and based on trial-and-error (in other words, not accurate!).

WHERE logs.user_id is not null
and logs.web_application_context_type = 'Course'
and (logs.web_application_action not IN
('jwt_token',
'activity_stream_summary',
'item_sequence',
'show_revision',
'ping',
'content_tag_assignment_data',
'progressions','permissions',
'enabled_features',
'index',
'show_front_page',
'launch_definitions',
'item_redirect')
OR logs.url like '%/modules?%' OR logs.url like '%/modules' or logs.url like '%/announcements')
and logs.web_application_controller not like '%_api%'
and logs.url not like '%/api%'

Unanswered Questions

EDUCAUSE Insights: Data and Decision-Making
EDUCAUSE is full of inspiration, whether you’re onsite or following from afar. Join the evolution by sharing what sparks your curiosity, the questions you have, and the insights you discover. We'll learn from each other, expand our networks, and strengthen our Community. Your perspective matters, and the more we share,…
Problems with GraphQL in User Pageviews reports
Prompted by the most recent Canvas release notes, we looked into the Page Views feature, and we were happy with the new documented changes. However, we notice that more user actions are done by GraphQL API, which does not provide details in the page view report, either in the GUI or in the CSV file download. We and most…
Metrics Easy Button
I am currently teaching five courses this semester with more than 70 students. To track their progress, I currently have to review each course and section individually, which is very time-consuming. Ideally, I would like to consolidate key information for all students into a single Excel file—showing class attendance…
Analytics API / Metrics calculation logic
Hello, We are currently using the Analytics API provided in the Instructure Developer Documentation Portal to retrieve certain metrics-related data. We would like to know if it’s possible to replicate these datasets using the Canvas Data Platform (CDP), instead of relying on API calls. For example, could you please advise…
CD2: Enhanced Rubrics
Is there a way to tell via CD2 if a course is using Enhanced Rubrics? I didn't see it in the entity diagram for Rubrics table, but I might have missed something... Thanks Ian