08NovA good url regular expression? (repost)

I’m moving this post from http://geekswithblogs.net/casualjim/archive/2005/12/01/61722.aspx

I started out blogging on geeks with blogs but I can’t allow comments there anymore or I get too much spam, so I’m moving the post from there to this place.  Various people have contributed through the comments in the other blog post. So here I have better control over the spam and can open the comments again.

I have been looking for a good first layer of validating an url to see if it is valid.

For checking the format of the url it seems to me to be the most logical approach to use regular expressions. Up until now I always discarded them as being to “geeky”, meaning i don’t consider it my life’s biggest goal to be typing (/?[]\w) all day long (so why did i become a programmer, aaaah yes to make life easier for other people)

Anyway.. to find a good regular expression to that validates urls not url domains. One that doesn’t allow spaces in the domainname and where the domain can be suffixed with the port number.  Also I need support for the ~/ paths

This is what I came up with.. if somebody as a better idea… or finds a mistake please let me know.. Always happy to learn something new.

^(((ht|f)tps?\:\/\/)|~/|/)?([a-zA-Z]{1}([\w\-]+\.)+([\w]{2,5})(:[\d]{1,5})?)/?(\w+\.[\w]{3,4})?((\?\w+=\w+)?(&\w+=\w+)*)?

I was a bit quickly in using this regex. Simeon pilgrim indicated that the ftp urls won’t validate when you add a username and a password.

I don’t really need to validate ftp so I should have removed the ftp protocol from the list of choices.  I need this just to validate urls for weblinks and the link element in an rss feed.  When I need them for ftp I will post the ftp version.. but for now I don’t have time to spend on elaborating the regex.

Anyway here is the right one : ^(http(s?)\:\/\/|~/|/)?([a-zA-Z]{1}([\w\-]+\.)+([\w]{2,5}))(:[\d]{1,5})?/?(\w+\.[\w]{3,4})?((\?\w+=\w+)?(&\w+=\w+)*)?

A full url validation would include resolving names through dns or making a webrequest to the provided url to see if we get a 200 response. The only way to be sure is to test if it is there in my opinion.

Thanks Simeon.

And for those who really want the ftp validation : ^((ht|f)tp(s?)\:\/\/|~/|/)?([\w]+:\w+@)?([a-zA-Z]{1}([\w\-]+\.)+([\w]{2,5}))(:[\d]{1,5})?/?(\w+\.[\w]{3,4})?((\?\w+=\w+)?(&\w+=\w+)*)?

I am not sure about numbers in the username but I believe you can have a username of numbers alone.

Comments don’t seem to work on this blog engine.. so just send me a mail through the contact form. thanks

Two days later …

I discovered there is still a problem with my regular expressions… folders don’t get parsed.

I’ve solved the path issue, so now it should be finding all url’s

Expression:

^((ht|f)tp(s?)\:\/\/|~/|/)?([\w]+:\w+@)?([a-zA-Z]{1}([\w\-]+\.)+([\w]{2,5}))(:[\d]{1,5})?((/?\w+/)+|/?)(\w+\.[\w]{3,4})?((\?\w+=\w+)?(&\w+=\w+)*)?

Should parse the url below

http://hh-1hallo.msn.blabla.com:80800/test/test/test.aspx?dd=dd&id=dki

But not :

http://hh-1hallo. msn.blablabla.com:80800/test/test.aspx?dd=dd&id=dki

Update 29/11/2008:

Joe posted what seems to be a great regular expression in the comments

he tested it with the following urls:

http://www.google.com/search?q=good+url+regex&rls=com.microsoft:*&ie=UTF-8&oe=UTF-8&startIndex=&startPage=1

ftp://joe:password@ftp.filetransferprotocal.com

google.ru

https://some-url.com?query=&name=joe?filter=*.*#some_anchor

Expression:

^(?#Protocol)(?:(?:ht|f)tp(?:s?)\:\/\/|~/|/)?(?#Username:Password)(?:\w+:\w+@)?(?#Subdomains)(?:(?:[-\w]+\.)+(?#TopLevel Domains)(?:com|org|net|gov|mil|biz|info|mobi|name|aero|jobs|museum|travel|[a-z]{2}))(?#Port)(?::[\d]{1,5})?(?#Directories)(?:(?:(?:/(?:[-\w~!$+|.,=]|%[a-f\d]{2})+)+|/)+|\?|#)?(?#Query)(?:(?:\?(?:[-\w~!$+|.,*:]|%[a-f\d{2}])+=(?:[-\w~!$+|.,*:=]|%[a-f\d]{2})*)(?:&(?:[-\w~!$+|.,*:]|%[a-f\d{2}])+=(?:[-\w~!$+|.,*:=]|%[a-f\d]{2})*)*)*(?#Anchor)(?:#(?:[-\w~!$+|.,*:=]|%[a-f\d]{2})*)?$

Update 8/11/2009:

Expression:

^(?#Protocol)(?:(?:ht|f)tp(?:s?)\:\/\/|~\/|\/)?(?#Username:Password)(?:\w+:\w+@)?(?#Subdomains)(?:(?:[-\w]+\.)+(?#TopLevel Domains)(?:com|org|net|gov|mil|biz|info|mobi|name|aero|jobs|museum|travel|[a-z]{2}))(?#Port)(?::[\d]{1,5})?(?#Directories)(?:(?:(?:\/(?:[-\w~!$+|.,=]|%[a-f\d]{2})+)+|\/)+|\?|#)?(?#Query)(?:(?:\?(?:[-\w~!$+|.,*:]|%[a-f\d{2}])+=?(?:[-\w~!$+|.,*:=]|%[a-f\d]{2})*)(?:&(?:[-\w~!$+|.,*:]|%[a-f\d{2}])+=?(?:[-\w~!$+|.,*:=]|%[a-f\d]{2})*)*)*(?#Anchor)(?:#(?:[-\w~!$+|.,*:=]|%[a-f\d]{2})*)?$

There is a wave for this regex:

https://wave.google.com/wave/?pli=1#restored:wave:googlewave.com!w%252BsFbGJUukA


  1. 1 Beni29 Nov 2009

    .asia ?

  2. 2 wilma30 Nov 2009

    Nice work. But this regex doesn’t cover IP-Addresses as hostnames. Unfortunately i was unable to fix it in a good way.

  3. 3 Manasto03 Dec 2009

    um…. i think you would absolutely be safe with something like (?:[a-zA-z}{2,8}) for Top Level Domains I don’t know what the max is, but i know the minimum is 2 characters for a dn.

  4. 4 Esben stien05 Dec 2009

    Well, this is not a general URL regular expression

    I’ve never tried to create one myself, cause I believe the IETF or some friends of them has already done this.

    Btw, something like http/ftp should not be included in the regex. as long as it follow a general format of : it’s a URL

  5. 5 ivan05 Dec 2009

    I think you missed the part of the post where it said what the purpose was. I’m sorry if URL isn’t used in the exact definition of the abbreviation URL. But lets not dwell on that.

  6. 6 testestest07 Dec 2009

    what about domains with asian or russian characters? i think now there are even chinese or korean top level domains possible, for example “中国”…

  7. 7 smb07 Jan 2010

    Don’t care about russian and asian characters. Before you validate an URL, you have to convert it to PunyCode, which only uses A-Z, 0-9 and -. So “中国” ain’t in URLs

  8. 8 petrelevich05 Feb 2010

    What would you say about this?

    ^(?:(?:(?:http[s]?):\/\/)|(?:www.))(?:[-_0-9a-z] .) [-_0-9a-z]{2,4}[:0-9]*[\/]*$

  9. 9 J. Benitez07 Feb 2010

    There’s no browser that can handle it

  10. 10 matapult08 Feb 2010

    Nope, didn’t work for me. I tried this ultra-basic implementation in python

    —————————————————————
    #!/usr/bin/env python

    import re
    line =”this is an example of a url that someone might write in some text http://www.google.com test test test”
    urls = []

    urls.append( re.findall( r”^(?#Protocol)(?:(?:ht|f)tp(?:s?)\:\/\/|~\/|\/)?(?#Username:Password)(?:\w :\w @)?(?#Subdomains)(?:(?:[-\w] \.) (?#TopLevel Domains)(?:com|org|net|gov|mil|biz|info|mobi|name|aero|jobs|museum|travel|[a-z]{2}))(?#Port)(?::[\d]{1,5})?(?#Directories)(?:(?:(?:\/(?:[-\w~!$ |.,=]|%[a-f\d]{2}) ) |\/) |\?|#)?(?#Query)(?:(?:\?(?:[-\w~!$ |.,*:]|%[a-f\d{2}]) =?(?:[-\w~!$ |.,*:=]|%[a-f\d]{2})*)(?:&(?:[-\w~!$ |.,*:]|%[a-f\d{2}]) =?(?:[-\w~!$ |.,*:=]|%[a-f\d]{2})*)*)*(?#Anchor)(?:#(?:[-\w~!$ |.,*:=]|%[a-f\d]{2})*)?$”, line ) )

    print urls
    —————————————————

    and it didnt find anything, not even the basic google.com :(

  11. 11 Rene15 Feb 2010

    I tried it with PHP and works flawlessly, I tried like 8 different ones from regexlibrary.com and this one killed them all. I’m just wondering if it will work with JS, I’ll keep you guys posted…

  12. 12 zerkms16 Feb 2010

    what about “&” in urls?

  13. 13 zerkms16 Feb 2010

    in the last comment i mean “& amp;” (without space)

  14. 14 Mina23 Feb 2010

    Thanks a lot for this wonderful article

  15. 15 Alexey Novikov14 Mar 2010

    I’ve enhanced it. Now it works with IPs
    Even if you write 1.01.001.000

    ^(?#Protocol)(?:(?:ht|f)tp(?:s?)\:\/\/|~\/|\/)?(?#Username:Password)(?:\w :\w @)?((?#Subdomains)(?:(?:[-\w] \.) (?#TopLevel Domains)(?:com|org|net|gov|mil|biz|info|mobi|name|aero|jobs|museum|travel|[a-z]{2}))|(?#IP)((\b25[0-5]\b|\b[2][0-4][0-9]\b|\b[0-1]?[0-9]?[0-9]\b)(\.(\b25[0-5]\b|\b[2][0-4][0-9]\b|\b[0-1]?[0-9]?[0-9]\b)){3}))(?#Port)(?::[\d]{1,5})?(?#Directories)(?:(?:(?:\/(?:[-\w~!$ |.,=]|%[a-f\d]{2}) ) |\/) |\?|#)?(?#Query)(?:(?:\?(?:[-\w~!$ |.,*:]|%[a-f\d{2}]) =?(?:[-\w~!$ |.,*:;=]|%[a-f\d]{2})*)(?:&(?:[-\w~!$ |.,*:]|%[a-f\d{2}]) =?(?:[-\w~!$ |.,*:;=]|%[a-f\d]{2})*)*)*(?#Anchor)(?:#(?:[-\w~!$ |.,*:;=]|%[a-f\d]{2})*)?$

  16. 16 Alexey Novikov14 Mar 2010

    By the way, there can be slashes after the anchor symbol (octothorp or dies or sharp). For example, in some gmail links. So if we do not to mark such URLs as invalid, we should include slash in the last part of this regexp:


    (?#Anchor)(?:#(?:[-\w~!$ |/.,*:;=]|%[a-f\d]{2})*)?$


Recent Flickrs

    Blogroll

    Recent Listening

    Scrobbler