Some time ago I wrote a program to compute an index for a course by walking the course pages and identifying key terms in the page, collecting all of the figure and table captions, all of the text that has been tagged as being in a language other than English, etc.
find_keyords_phrase_in_files.py and create_page_from_json.py
see the heading "Making an index" at https://github.com/gqmaguirejr/Canvas-tools and details of using it can be found at https://canvas.kth.se/courses/11/pages/indexing-a-course?module_item_id=232285
The result is a wikipage with entries of the form:
Note that the anchor HREF is to a relative location the "../" gets replace by the browser with the prefix for the page (https://canvas.kth.se/courses/21521) yielding the full URL https://canvas.kth.se/courses/21521/modules/items/316319
I had to use these relative HREFs to reduce the index for the course down to 3 pages due to the limited size of wikipages. This works perfectly well for the student in the course.
If I run the link validator from the page that says:
Course Link Validator
The course link validator searches course content for invalid or unreachable links and images.
The link validator reports:Found 14,610 broken links
Nearly all of the claimed broken links do in fact work.
When this problem was reported by my institution to Instructure the responses were:
Instructure support responded to the first message about this with:
>
> Thank you for contacting Canvas support! I'm sorry to hear you are
> experiencing some issues in Canvas. I would be happy to help you with
> that.
> Our Link validator often fails on valid links when content is copied.
> There is not a workaround for this. Canvas is aware of this issue and
> we have a team investigating this. I have attached your case to their
> projects so you will get updates on this issue. Please let us know if
> you have any additional questions.
Their second response was:
So it does appear that the instructor is right and there is an issue with the href and the link validator. Unfortunately our link validator is extremely literal in that it needs the full link and for it to go directly to the page or it flags the link as broken. In this case if we look at the HTML we can see the href reads ../modules/items/316861 for one of the links and this is valid this link works, the browser resolves the ../ or relative path by just adding it to the end of the current URL. This is also why in the link validator its getting flagged is because the link validator can't find the course for the relative since it doesn't work the same as the browser.
A link will also get flagged if there is a re-direct. If you have a working link for an article that used to live at "workinglink.com" will say, but they were bought out by "welikebrokenlinks.com" and all articles moved there, even though the browser knows to resolve this with a redirect the link validator doesn't and will flag it. The Link Validator is good for getting a general idea of which links may be broken but its not 100% accurate.
Looking at the code the ERB for the link validator gets invoked via ./app/views/courses/link_validator.html.erb
<%<BR />content_for :page_title, join_title(t(:page_title, "Course Link Validator"), @context.name)
js_env :validation_api_url => api_v1_course_link_validation_url(@context)
js_bundle :course_link_validator
css_bundle :course_link_validator
%>
So in fact it knows the content in which it is to be validating the links.
Eventually the ruby code in ./lib/course_link_validator.rb gets invoked and for wikipages it does:
# Wiki pages
self.course.wiki_pages.not_deleted.each do |page|
find_invalid_links(page.body) do |links|
self.issues << {:name => page.title, :type => :wiki_page,
:content_url => "/courses/#{self.course.id}/pages/#{page.url}"}.merge(:invalid_links => links)
end
end
Thus it invokes the following two functions:
# pretty much copied from ImportedHtmlConverter
def find_invalid_links(html)
links = []
doc = Nokogiri::HTML(html || "")
attrs = ['href', 'src', 'data', 'value']
doc.search("*").each do |node|
attrs.each do |attr|
url = node[attr]
next unless url.present?
if attr == 'value'
next unless node['name'] && node['name'] == 'src'
end
find_invalid_link(url) do |invalid_link|
link_text = node.text.presence
invalid_link[:link_text] = link_text if link_text
invalid_link[:image] = true if node.name == 'img'
links << invalid_link
end
end
end
yield links if links.any?
end
# yields a hash containing the url and an error type if the url is invalid
def find_invalid_link(url)
return if url.start_with?('mailto:')
unless result = self.visited_urls[url]
begin
if ImportedHtmlConverter.relative_url?(url) || (self.domain_regex && url.match(self.domain_regex))
if valid_route?(url)
if url.match(/\/courses\/(\d+)/) && self.course.id.to_s != $1
result = :course_mismatch
else
result = check_object_status(url)
end
else
result = :unreachable
end
else
unless reachable_url?(url)
result = :unreachable
end
end
rescue URI::Error
result = :unparsable
end
result ||= :success
self.visited_urls[url] = result
end
unless result == :success
invalid_link = {:url => url, :reason => result}
yield invalid_link
end
end
So we see that they did think about relative links (in the case of ImportedHtmlConverter.relative_url) - but not well enough.
