> ## Content Index
> Fetch the complete content index at: https://serpapi.com/blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# URL Normalization caused a bug in our service
- URL: https://serpapi.com/blog/url-normization-caused-a-bug-in-our-service/
- Published: 2023-12-29T01:16:37.000Z
- Updated: 2024-01-03T23:51:20.000Z
- Author: Terry Tan
- Tags: URL Normalization, Development

Recently in [SerpApi](http://serpapi.com/), our customers faced an issue in which using a query that contains some form of Unicode wouldn’t work, for example `synergy²-madison-3`, noticed the `²`. This is supported by the browsers, navigate to [https://www.yelp.com/biz/synergy²-madison-3](https://www.yelp.com/biz/synergy%C2%B2-madison-3) in Chrome should work just fine, but there is an issue with the underlying library that we use. 

> This happened to [Yelp Place API](https://serpapi.com/yelp-place) and we have made a fix on the issue, feel free to checkout our [playground](https://serpapi.com/playground?engine=yelp%5Fplace&place%5Fid=synergy%C2%B2-madison-3).

Our [Yelp Place API](https://serpapi.com/yelp-place) takes `place_id` parameter and forms a complete HTTP URL, for example, `https://www.yelp.com/biz/synergy²-madison-3` where `place_id` is `synergy²-madison-3`. We then made the HTTP call to download the data, but this time it didn't work as expected. The response was always `301 - Moved Permanently`. It might be wise to enable `follow redirect` in the [http](https://github.com/httprb/http) library (we are on version `v4`), but that would end up in `EndlessRedirectError`.

After digging through the library code, turns out that, http library normalizes the URL before requesting it for various reasons. Thus, `https://www.yelp.com/biz/synergy²-madison-3` became `https://www.yelp.com/biz/synergy2-madison-3` after normalization, it is no longer the same URL. Yelp is smart enough to redirect `.../biz/synergy2-madison-3b` back to `.../biz/synergy²-madison-3`, thus the `301 - Moved Permanently` response and that also explains the `EndlessRedirectError` after we enabled `follow redirect`, because the http library will normalize again the URL to `.../biz/synergy2-madison-3b` before requesting.

http `v5` uses a [more conservative URL normalization](https://github.com/httprb/http/pull/758) and it resolves our issue, which Unicode character is percent-encoded `.../biz/synergy%C2%B2-madison-3` instead of changing its meaning.

Underlying the http library is [addressable](https://github.com/sporkmonger/addressable) where the actual normalization is done (we were on `v2.7`). [The update](https://github.com/sporkmonger/addressable/commit/1998e06c81d813eac30409497b04fc6d6f19d231#diff-831b213e8c4e581a6b683edba10e28c446b5b9db581fbdca650286aa7a486db7) that fixed our issue is on the version `v2.8.2`, it changed the encoding from `NFKC` to `NFC`. Here is the comparison between them:

> Encoding `synergy²-madison-3` in `NFKC` ⇒ `synergy2-madison-3`  
> Encoding `synergy²-madison-3` in `NFC` ⇒ `synergy²-madison-3`

There is a lot of information about which encoding is used for URL normalization and the reasons behind them, I am not an expert so I can't share more about it. But clearly, using `NFC` resolved our issue. The final solution we went for is to upgrade addressable to the latest `v2.8.6` as it only involves a single dependency upgrade.

## Why is URI normalization needed?

There are a lot of reasons why normalization is important and this topic can get very complicated. As for Google, URI normalization can help to prevent duplicated content. If the same content is accessible via multiple URLs, it can dilute the page's ranking or lead to confusion about which version is the canonical one. URL normalization helps to prevent such duplication.

In terms of security, URI normalization prevents attacks set up by people with intention. URI normalization helps in identifying potential security vulnerabilities. Malicious actors might use variations of URIs to bypass security controls or deceive users. By normalizing URIs, it's easier to detect and block such attempts.

Normalizing URIs can also help in efficient caching and storage. If multiple representations of the same resource are cached or stored separately, it can lead to the wastage of resources. Normalization ensures that only one canonical representation is used for a given resource.

## **Conclusion**

It is interesting to get to know implementations like URL Normalization. It is something we might not often think about or even take for granted, but I believe it is one of the important building blocks of the web for security, efficiency, and other critical reasons.