New SEMrush Database – May 2019 Data

[rochon.a1.119]

Tracking queue
Early testing of the new crawler revealed that the crawl queue was not properly handling the amount of data collected. We attempted to address this issue by increasing the hardware capacity, but this was not enough, so we developed a new, more efficient crawl queue.

Seeding
To provide our crawler with a quality initial selection, we created a queue of Google's top 100 URLs with the goal of getting 450 million keywords from our organic research tool .

With this idea we made our database relevant from the beginning.

Storage
An increasing collection of data obviously requires more storage space, so we had to triple the size of our server.

Measuring our growth
To find out exactly where we stand as a backlink provider, we decided to measure our strength against the best: Majestic , Ahrefs , and Moz .

We will explain the methodology shortly but first, let's evaluate our development progress over the past six months.

Looking at the relationship between the top four SEO tools on the market, you can see that we have made a significant leap forward.

Here's what the comparison looked like in October 2018:


New SEMrush Backlinks Database – October 2018 afghanistan phone number list Data
This is what this comparison looks like today.



Important! How to interpret the graphs correctly?
We assume two cases:

If SEMrush is represented by 1 and competitors by higher values, then in this case we set the volume of data from other providers relative to our data. A simple example: we have an apple. Competitor A has 2 apples and competitor B has 3 apples. Compared to us, competitor A has 2 times as many and competitor B 3 times as many apples. In this case, our competitors are better than us.

If SEMrush is represented by a 1 and competitors by lower values, the procedure would be the same as in the previous point with the difference that this time the competitors perform worse than us.