How To Build Links Detector That Making Links in Your Book Clickable

🤷🏻‍ The Problem

I bought a printed book about Machine Learning recently and while I was reading through the first several chapters I've encountered many printed links in the text that looked like

https://tensorflow.org/ 

or  

https://some-url.com/which/may/be/even/longer?and_with_params=true

.

I saw all these links, but I couldn't click on them since they were printed (thanks, cap!). To visit these links I needed to start typing them character by character in the browser's address bar, which was pretty annoying and error-prone.

💡 The Solution

So, I was thinking, what if, similarly to QR-code detection, we will try to "teach" the smartphone to (1) detect and (2) recognize printed links for us and to make them clickable? This way you would do just one click instead of multiple keystrokes. The operational complexity of "clicking" the printed links goes from 

O(N)

 to 

O(1)

.

This is exactly what I've tried to achieve by making a Links Detector app. It makes you do just one click on the link instead of typing the whole link manually character by character.

I came up with a custom dataset of

120

photos of book pages that contained links in it. I used TensorFlow 2 Object Detection API to train a custom object detector model to find positions and bounding boxes of the sub-strings like 

https://

 in the text image (i.e. in smartphone camera stream). You may found the details of the training in 📖 👆🏻 Making the Printed Links Clickable Using TensorFlow 2 Object Detection API long read article.

The text of each link (right continuation of 

https://

 bounding box) was recognized by using Tesseract library.

🚀 Launch Links Detector demo from your smartphone to see the final result.

📝 Open links-detector repository on GitHub to see the complete source code of the application.

⚠️ Limitations

Currently, the application is in experimental Alpha stage and has many issues and limitations. So don't raise your expectations level too high until these issues are resolved 🤷🏻‍.

⚙️ Technologies

Links Detector is a pure frontend React application written on TypeScript. Links detection is happening right in your browser without the need of sending images to the server.

Links Detector is PWA (Progressive Web App) friendly application made on top of a Workbox library. While you navigate through the app it tries to cache all resources to make them available offline and to make consequent visits much faster for you. You may also install Links Detector as a standalone app on your smartphone.

Links detection and recognition happens by means of TensorFlow and Tesseract.js libraries which in turn rely on WebGL and WebAssembly browser support.

Also published at https://towardsdatascience.com/making-the-printed-links-clickable-using-tensorflow-2-object-detection-api-be42bd65488a