Improve error handling on LinkCrawlWorker (#5250)

* Improve error handling on LinkCrawlWorker

* Ignore TimeoutError and InvalidURIError too
* Record errors to debug log
* Enable dead job queue on LinkCrawlWorker

Since most of acceptable errors were already ignored, only our side issue should go to dead job queue.

* Ignore all http gem errors

diff --git a/app/services/fetch_link_card_service.rb b/app/services/fetch_link_card_service.rb
index 4acbfae7a04f09779309430785426fd1e04514d8..cf3d7868303d5c5b77d03c13868681c0702dffdc 100644 (file)
--- a/app/services/fetch_link_card_service.rb
+++ b/app/services/fetch_link_card_service.rb
@@ -27,7 +27,8 @@ class FetchLinkCardService < BaseService
     end
 
     attach_card if @card&.persisted?
-  rescue HTTP::ConnectionError, OpenSSL::SSL::SSLError
+  rescue HTTP::Error, Addressable::URI::InvalidURIError => e
+    Rails.logger.debug "Error fetching link #{@url}: #{e}"
     nil
   end
 

diff --git a/app/workers/link_crawl_worker.rb b/app/workers/link_crawl_worker.rb
index 834b0088bdfd1993eb4cbbf68cbec2ac9d3cd624..b3d8aa26467dfc501aabae702bd88e7da7e2879f 100644 (file)
--- a/app/workers/link_crawl_worker.rb
+++ b/app/workers/link_crawl_worker.rb
@@ -3,7 +3,7 @@
 class LinkCrawlWorker
   include Sidekiq::Worker
 
-  sidekiq_options queue: 'pull', retry: false
+  sidekiq_options queue: 'pull', retry: 0
 
   def perform(status_id)
     FetchLinkCardService.new.call(Status.find(status_id))