URL Shortner — How to build one?
A URL shortener is a web service or tool that takes a long and unwieldy Uniform Resource Locator (URL) and converts it into a shorter, more manageable form. The primary purpose of a URL shortener is to make long URLs more user-friendly, shareable, and easier to remember.
It’s important to note that while URL shorteners offer many benefits, they do have some drawbacks, such as potential link obfuscation (users can’t easily see where the link leads) and concerns about privacy and the longevity of short links. As a result, the use of URL shorteners has evolved, and some services offer features like link previews and the ability to set expiration dates for short links to address these issues. Regardless, a URL shortners are widely used to make it more convenient for the users to track those URLs when they are surfing the internet.
In this blog, I’ll be explaining how you can create your own custom URL shortner instead of using a 3rd party package to do so.
The Basics
In order to build a URL shortner there’s some required background knowledge that is essential to dive deep into building one. In order to build a URL shortner, you can use a hash table, but the issue with hash tables is that when the data set becomes large, it takes up a lot of memory and the searching operation becomes very time consuming. So, in order to get rid of that, it is suggested to use a relational database like MySQL or PostgreSQL where we will create a single table that can store “id”, “longUrl” and the corresponding “shortUrl”. I will be using a SQL database for this in this article.
Now, the requirements of our URL shortner are:
- Take the URL in input and return short URL
- If a short URL is given, redirect the user to the original URL
- The generated short URL should contain only alphabets (a-z), (A-Z) and numbers (0–9)
- If there already exists a short URL corresponding to the input long URL, return the same short URL instead of generating a new one
Hash Function
We will require a hashing function that could take the input url and create a unique hash corresponding to it so that we can append that hash value with our base URL (http://www.shortify.com) and generate a unique short URL for every new URL that is provided.
Now, as we discussed, the hash value can contain alphabets (a-z), (A-Z) and numbers (0–9), which makes a total of 62 characters at our disposal. Suppose we are making our service such that it can support 365 billion unique URLs over it’s entire life. So, doing a little calculation can tell us how many characters should our hash value contain in order to create 365,000,000,000 unique hash values.
Back of the envelope calculations
If we have 62 characters at our disposal, it means that at every character of the hash value we can insert 62 different values, therefore for ’n’ length of hash value, we can have ‘62^n’ unique combinations. Taking n=7, we will get approximately ~3.5 trillion unique hash values, meaning that we can produce ~3.5 trillion unique short URLs, which is enough for now.
Which Hashing Algorithm to use?
You can definitely use the common hashing algorithms like CRC32, MD5, SHA-2, etc, but here’s a table depicting the length of hash values produced by some of these algorithms:
As you can see, the shortest possible hash value that is produced by CRC32 is also >7 characters long. Therefore, we won’t be using any of these algorithms for our use. Then hey, what will we use?
Base 62 Conversion
Base 62 is a way of using 62 characters for encoding. The mappings are: 0–0, …, 9–9, 10-a, 11-b, …, 35-z, 36-A, …, 61-Z, where ‘a’ stands for 10, ‘Z’ stands for 61, etc.
1115710 = 2x62^2 + 55x62^1 + 59x62^0 = [2,55,59] -> [2,T,X] in base 62 representation
This is a pretty convenient way for us to produce unique hash values. The basic flow is:
- Create a unique ID for every input long URL
- Do base 62 conversion for that UUID to create a unique hash value
- Append the produced hash value in the input long URL
- Store the long url along with the generated short URL in the database
Building the URL Shortner
Step-1: Setting up the directory and installing the dependencies
Go to the directory where you want to build the URL Shortner and then run the following commands:
My directory at the end of building the entire URL Shortner looks like this:
npm init --y
npm install express cors dotenv mysql2 uuidv4
npm install nodemon -DStep-2: Setting up a MySQL database connection (database.js)
const mysql = require('mysql2');
// create a new mysql connection
const connection = mysql.createConnection({
host: 'localhost',
database: 'URL_Shortner',
user: 'root',
password: 'mypassword123'
})
// connect to the mysql database
connection.connect((error) => {
if(error){
console.error("Error connecting to mysql database server", error);
}else{
console.log("Connected to MySQL Dabatabse!");
}
})
module.exports = connection;Step-3: Setting up the Express Server
Once you have written the connection code for your database, it’s time to finally build our server.
For the sake of simplicity, I have made the server using two files, in one (index.js) I have declared the required dependencies, whereas I have written the code for actual routes in another file(express-app.js). Follow the code below to see how I have made them.
index.js -
const express = require('express');
const {PORT} = require('./utils');
const expressApp = require('./express-app');
const database = require('./database');
const StartServer = async () => {
const app = express();
await expressApp(app, database);
app.listen(PORT, () => {
console.log(`Listening on port ${PORT}`);
})
.on('error', (err) => {
console.log(err);
process.exit();
})
}
StartServer();express-app.js -
const express = require('express');
const cors = require('cors');
const connection = require('./database');
const { v4: uuidv4 } = require('uuid');
const referenceTable = require('./referenceTable');
module.exports = async (app, database) => {
//middlewares
app.use(express.json({ limit: '1mb'}));
app.use(express.urlencoded({ extended: true, limit: '1mb'}));
app.use(cors({origin: "*"}));
app.use(express.static(__dirname + '/public'))
// APIs
app.get('/shortify/:hashValue', (req, res) => {
const shortUrl = 'http://localhost:8000/shortify/' + req.params.hashValue;
// find the ogUrl corresponding to this short url in our database
connection.query('SELECT * FROM Container WHERE shortUrl=?', [shortUrl], (err, rows) => {
if(err) console.log(err);
else{
const ogUrl = rows[0].longUrl;
res.status(200).json({
"success": true,
"ogUrl": ogUrl
})
//res.redirect(ogUrl);
}
})
})
app.post('/shortify/postUrl', (req, res) => {
const ogUrl = req.body.url;
// check if a short url corresponsing to this URL exists already
connection.query(`SELECT * FROM Container WHERE longUrl=?`, [ogUrl], (err, rows) => {
if(err){
console.log(err);
}else{
if(rows.length === 0){
// means that the given url is not present in the database
// generate the short-url for this input url
// Step-1: Generate a unique ID for the new URL entry
const uuid = uuidv4();
var numericID = 1;
for(let i=0; i<uuid.length; i++){
let ch = uuid[i];
let val = ch.charCodeAt(0);
if(val >= 48 && val <= 57){
numericID += (val - 48);
}else if(val >= 65 && val <= 90){
numericID += (val - 65 + 11);
}else if(val >= 97 && val <= 122){
numericID += (val - 97 + 73);
}
}
const salt = Math.ceil(Math.random()*100)*23*7;
numericID = numericID * salt;
// Step - 2: Base 62 conversion
var genHashVal ="";
let dummyId = numericID;
while(dummyId > 0){
const rem = dummyId % 62;
genHashVal += referenceTable[rem];
dummyId = Math.floor(dummyId/62);
}
// we have generated the short hashValue
const hashValue = genHashVal;
// Step-3: Generate your own short url from this hashed value
var shortUrl = "http://localhost:8000/shortify/" + hashValue;
// Step-4: Save this shortUrl with the ogUrl in the db
connection.query("INSERT INTO Container (longUrl, shortUrl) VALUES (?)", [[ogUrl, shortUrl]], (err, rows) => {
if(err){console.log(err)}
else{console.log(rows)}
})
res.status(200).json({
"message": "Inserted the new URL",
"shortUrl": shortUrl,
"longUrl": ogUrl
})
}else{
res.status(200).json({
"message": "URL already shortified!",
"shortUrl": rows[0].shortUrl,
"longUrl": ogUrl
})
}
}
})
})
}Now you can simply run the server by using “nodemon index.js” while being in the same directory as the index.js file.
Step-4 : Testing our URL Shortner
In order to test our URL shortner, you can use POSTMAN to send get and post requests to the two endpoints we have built.
As you can see in the image above, the server generates a short URL and sends it back along with the actual URL.
This is how you can build a URL Shortner and use it wherever you want.
What’s Next?
Well, now that you know how to build a URL Shortner, you can embed it in your application to generate simple and clean URLs that users can use.
You can get the entire code for this on my github profile over here.
Feel free to get in touch with me on LinkedIn, or follow me on Twitter. 🐘
