jarink releases changelog

🌼 brokenlinks: refactoring the logic, simplify the code

Previously, we made the scan logic to run in multiple goroutine with one channel to push and consume the result and another channel to push and pop link to be processed. The logic is a very complicated code, making it hard to read and debug.

These changes refactoring it to use single goroutine that push and pop link from/to a slices, as queue.

Another refactoring is in where we store the link. Previously, we have [jarink.Link], [brokenlinks.Broken], and [brokenlinks.linkQueue] to store the metadata for a link. These release unified them into struct [jarink.Link].

🌱 brokenlinks: print the progress to stderr

Each time the scan start, new queue add, fetching start, print the message to stderr. This remove the verbose options for better user experience.

🌼 brokenlinks: improve fetch logging and decrease timeout to 10s

When fetching, print log after the fetch completed. If success, print the URL along with HTTP status code. If fail, print the URL along with the error.

The timeout now reduce to 10 seconds to prevent long delay when working with broken website.

🌼 brokenlinks: mark the link in queue as seen with status code 0

This is to fix double URL being pushed to queue.

Given the following queue and its parent,

Before scanning the second "/brokenPage" on parent page "/page2.html", check if its seen first to get the status code before we run the scan. This allow jarink report "/brokenPage" as broken link for both pages, not just in "/index.html".

🌼 brokenlinks: skip parsing non-HTML page

If the response Content-type return other than "text/html", skip parsing the content and return immediately.

We also skip processing "mailto:" URL.

🌼 brokenlinks: make link that return HTML always end with slash

If parent URL like "/page" return the body as HTML page, the URL should be end with slash to make the relative links inside it works when joined with the parent URL.

🌱 brokenlinks: store the anchor or image source in link

In the struct Link, we add field Value that store the href from A element or src from IMG element. This allow us to debug any error during scan, especially joining path and link.

🌼 brokenlinks: fix possible panic in markAsBroken

If the Link does not have parentUrl, set the parent URL using the link URL itself. This only happened if the target URL that we will scan return an error.

🌼 brokenlinks: fix infinite loop on unknown host

On link with invalid domain, it should break and return the error immediately.

🌱 brokenlinks: add option to ignore list HTTP status code.

When link known to have an issues, one can ignore the status code during scanning broken links using "-ignore-status" option.

🌱 brokenlinks: add option "insecure".

The "-insecure" option does not report an error on server with invalid certificates.

🌱 brokenlinks: implement caching for external URLs.

Any successful fetch on external URLs will be recorded into jarink cache file, located in user’s cache directory. For example, in Linux it would be $HOME/.cache/jarink/cache.json. This help improve the future rescanning on the same or different target URL, minimizing network requests.

🌼 brokenlinks: reduce the number of goroutine on scan.

Previously, each scan run on one goroutine and the result is pushed using one goroutine. This makes one scan of link consume two goroutine.

This changes the scan function to return the result and push it in the same goroutine.

The first release of jarink provides the command "brokenlinks", to scan for broken links. The output of this command is list of page with its broken links in JSON format.

This command accept the following options,