2022-04-16
I have a vision, new to me, but quite possibly not new to some other people, of how the web could work in a different way, much in line with my ideas for wikis. Handling backlinks could be done in a very sophisticated way, probably through the web server itself rather than something built in a scripting language. Naturally though, prototypes would be built in a scripting language.
All http(s) requests are scanned for a referring page. If there isn't a referer (canonized misspelling of ‘referrer’), obviously no back link is made.
If there is a referer, then the referring page is scanned to check that it really has a link.
If there is no link there, then the referer must be spoofed. A good side effect of this is that the request is delayed. However there is a load on one's own server, making it more prone to DDOS attacks.
If the referer has a proper link, then it goes into a decision process (I really should draw this as a flowchart, and write it in pseudocode)
There is a whitelist of whole domains that are trusted, and a list of backlinked pages is kept from this site. If the referer is already backlinked from the linked page, no action is needed; if it is already backlinked from another page then the backlink is added to the linked page. If it is not already linked, the page is checked for a link, and if found the backlink is automatically added, with an appropriate relation. How this information is held depends in part on how backlink information is held by the site itself.
There is a blacklist of malicious sites, with a redirect directing incoming links from this site as referer to a 404; or a rejection page explaining what is wrong with the referring site.
There is a greylist of sites that have not been white- or blacklisted, but at least one page has been accepted for backlinking. Accepted pages from this site are kept in a list. If the referer is on this accepted list, the request is accepted similarly to the whitelist. If it is not on the list, the page is flagged for review.
If the referer domain is not on the black or white list, or the referer page is in a greylist domain but not already backlinked, the linking page and/or site is held for human review.
A site admin can add the whole domain to the black or white list. A new blacklist entry sets up a redirect as described above. A new whitelist entry also adds the page to the list of backlinks from that domain.
A site admin can add a backlink to that page and add the site to the grey list
For each backlinked page, either a dated copy or a hash is kept. At any convenient time, or at convenient intervals, a check may be done on the current page.
If the page has changed, this is flagged for human review.
If a copy has been kept, then the review is given as a diff.
If storage is plentiful, old dated copies can be retained.
If the page has gone away, backlinks are deleted.
When a human reviews a page for potential backlinking, that is also recorded, to be transparent and accountable if problems arise.
(added 2025-11-21)
A much simpler approach could be used to start with, or in cases where there was interest only in particular sister sites. In this case, there is no need for a blacklist or even a greylist.
The referer is scanned and checked, as before.
If the domain is whitelisted and the page has an actual link, a backlink is automatically added if not already present.
If the domain is whitelisted but the page has no link, then if there is a backlink it is removed. Other backlinks from that domain are also checked.
If all the others are fine, no action is needed.
Any pages now found not to link have their backlinks removed.
If none of the pages have a link, the domain is removed from the whitelist. This may happen if the domain has changed hands or no longer has relevant related information.
If the domain is not whitelisted and the page has an actual link, the
URL may be kept for later processing, if desired.
If there is no interest in extending backlinks, it can simply be ignored.
If there is potential interest in expanding the backlinking club, some kind of automatic assessment of quality and relevance can be done, and if the site passes this test, it is submitted to the owner for potential whitelisting.
If and when a site is added to the whitelist, an automatic search of the whole site is carried out, and any links are then backlinked appropriately.