I just wanted to share the presentation I gave last Friday at OWASP Latam Tour Buenos Aires. Many thanks to the organizers which let me speak, and made everything to make the event great!
Abstract:
IT companies that do heavy software development have been shifting their paradigm from a traditional monolithic waterfall development lifecycle to a fully heterogeneous 24/7 devops culture. This implies more software deployment and more code developed. The traditional security approach, besides not being enough, is clearly outdated and non-applicable. This talk will tell how MercadoLibre evolved to a DevOps company, how information security was perceived and tackled then and now, what challenges we faced, what we made to drive change to a 15 years old company’s mindset, and how we are transforming into a SecDevOps culture and the way we envision that culture of work.
Sunday, April 26, 2015
Wednesday, March 25, 2015
Fixing Open Redirect
Sometimes most common vulnerabilities are not that easy to fix or prevent, one of them can be "Open Redirect"
A short example is: you are shopping at an e-commerce, you visit an item and want to save that item for later, and click on "favourite". The sites redirects to the login; after the login you are redirected back to the item.
Keep in mind that the login and item page are not necessarily in the same domain.
(Example: youtube.com login is on google.com)
Note: More detail from OWASP can be found here
The easiest way to achieve this, is if the controller of authenticated actions, like "favourite" redirects to the login sending as a parameter de item page, so it know where to go back.
https://site.com/login?back_url=http://site.com/buy/itemX
Note: keep in mind that the back_url parametere must be URL Encoded.
A few of the risks of the abover solution are:
Abusing the trust a victim has in the domain so it clicks on a link that ends in a phishing or a malware download or a Browser exploit attempt.
Consider: If a site mitigates CSRF with Referer domain validation it can be faked with an Open Redirect.
Solution:
There are many solutions I'll describe just two.
1. URL HMAC Signing
In the "favourite" controller, do a 302 redirect to this URL:
http://site.com/login?back_url=http://site.com/buy/itemX&HMAC=af398bc82720a
example: HMAC(SHA2, Key, "http://site.com/buy/itemX") = af398bc82720a
NOTE: the result will be 160 bits long for SHA1 and 256 bits for SHA256, hex encoding will work just fine.
The Secret Key will have to be shared among the "login" and the controllers that perform de redirection to it.
Consideration: Don't use the Referer Header to generate the back_url, create the URL server side.
When the login receives the back_url, it recalculates the HMAC, if it matches with the one provided in the link it performs the redirection.
Advantage:
1. Can easily fix existing open redirects.
2. It's not vulnerable to HTTP Response splitting
3. HMAC is easy to use and readily available.
Disadvantage:
1. Needs to share a Secret key among many controllers, and potentially many Developers.
2. Needs to change all the controllers that perform redirections ("login" and the ones that generates the redirects ("itemX")
2. Validating domains with tested libraries
When you receive this parameter in the "login" controller:
back_url=http://site.com/buy/itemX
You can validate that the domain "site.com" is legit.
Note: use "endsWith", "contains" allows "site.com.hack.com". Since "getHost()" gets you also subdomains doing a straight comparison forces you to have a list of subdomains, instead of just domains. When using "endsWith" always include a dot "." at the beginning, if not, you would accept "http://hacker-site.com". If you want to accept "http://site.com" you have to add the or statement asking for equals.
Advantage:
1. You only have to make this validation in the redirection controllers, such as login
Disadvantage:
1. You have to keep an updated legit domain list.
2. If the library you used has a flaw you could be vulnerable.
3. If no library is available, making one with regex is extremely prone to error.
4. You have to canonicalize the input and do a validation of the characters, because the attacker might inject a end of line character, causing a header injection or worse, an HTTP Response Splitting.
Potential solutions that won't work or are prone to fail
1. In this situation OWASP first recommendation, avoid redirects, can't be easily achieved.
The second recommendation, don’t involve user parameters, can't be easily achieved neither.
One might think using "back_url=itemX" will solve the problem, the thing is that every section that is not an item that redirects to the login must have a mapping. This will turn out to be a mapping nightmare.
2. Presses "buy"
3. A javascript determines if the users is logged or not by watching a Boolean flag cookie. (without HTTPOnly of course)
4A. If the flag says it's logged in, the "buy" actions hits the "buy" controller.
4B. If the flag says it's not logged in, the "buy" actions, performs a POST to the "login" controller, and sends with it, the URL of the visited item, so It knows where to go back.
5B. The login page is displayed, and in a hidden field resides the URL to go back.
6B. If the login is successful it does a redirect to the URL stored in the hidden field, which was send along with the user and password in the POST.
The good thing about this solution is that you don't share any information between "buy" and "login" through the back-end, and you don't store anything in the back-end either.
The first thing you might think is Tampering with the Boolean flag cookie, but if the cookie is forged to look like logged out, and you are logged in, it will send you to the login, nothing wrong with it. If the cookie is forged to look like logged in when you are not, you will go to the "buy" controller, which will tell very quickly that you are not logged in, no harm there.
A potential attacker might create a site that only does a POST to the login, sending the a fake URL to go back, potentially a forged clone of the original login. When the victim does a successful login, it's redirected to the fake login, where it's asked for the user and password again. This way the attacker can exploit this solution.
The big problem is that the login can't be sure who is sending the URL to go back! Using the "Referer" header is not safe as told before, and it's not always there (HTTPS link from one domain to another).
There's probable more bad solutions than good ones, so keep in mind which one you use.
A short example is: you are shopping at an e-commerce, you visit an item and want to save that item for later, and click on "favourite". The sites redirects to the login; after the login you are redirected back to the item.
Keep in mind that the login and item page are not necessarily in the same domain.
(Example: youtube.com login is on google.com)
Note: More detail from OWASP can be found here
https://site.com/login?back_url=http://site.com/buy/itemX
Note: keep in mind that the back_url parametere must be URL Encoded.
A few of the risks of the abover solution are:
Abusing the trust a victim has in the domain so it clicks on a link that ends in a phishing or a malware download or a Browser exploit attempt.
Consider: If a site mitigates CSRF with Referer domain validation it can be faked with an Open Redirect.
Solution:
There are many solutions I'll describe just two.
1. URL HMAC Signing
In the "favourite" controller, do a 302 redirect to this URL:
http://site.com/login?back_url=http://site.com/buy/itemX&HMAC=af398bc82720a
HMAC is a signature of the back_url parameter. HMAC is very easy to use and available in most languages.
You have to decide wich hash algorithm to use. Use SHA2 if possible, but SHA1 will do the job as well.
You have to use a Secret Key, for this generate a 256 bit Secure Random.
example: HMAC(SHA2, Key, "http://site.com/buy/itemX") = af398bc82720a
NOTE: the result will be 160 bits long for SHA1 and 256 bits for SHA256, hex encoding will work just fine.
The Secret Key will have to be shared among the "login" and the controllers that perform de redirection to it.
Consideration: Don't use the Referer Header to generate the back_url, create the URL server side.
When the login receives the back_url, it recalculates the HMAC, if it matches with the one provided in the link it performs the redirection.
import javax.crypto.Mac; import javax.crypto.spec.SecretKeySpec; import java.io.UnsupportedEncodingException; import java.security.InvalidKeyException; import java.security.NoSuchAlgorithmException; import javax.xml.bind.DatatypeConverter; public class SignURL { public static void main(String[] args) throws Exception { System.out.println(hmacDigest("http://site.com/ItemX", "SecurelyGeneratedKeyOf256SecureRandomBits", "HmacSHA256")); } public static String hmacDigest(String msg, String keyString, String algo) { String digest = null; try { SecretKeySpec key = new SecretKeySpec((keyString).getBytes("UTF-8"), algo); Mac mac = Mac.getInstance(algo); mac.init(key); byte[] bytes = mac.doFinal(msg.getBytes("ASCII")); digest = DatatypeConverter.printHexBinary(bytes); } catch (UnsupportedEncodingException e) { } catch (InvalidKeyException e) { } catch (NoSuchAlgorithmException e) { } return digest; } }
Advantage:
1. Can easily fix existing open redirects.
2. It's not vulnerable to HTTP Response splitting
3. HMAC is easy to use and readily available.
Disadvantage:
1. Needs to share a Secret key among many controllers, and potentially many Developers.
2. Needs to change all the controllers that perform redirections ("login" and the ones that generates the redirects ("itemX")
2. Validating domains with tested libraries
When you receive this parameter in the "login" controller:
back_url=http://site.com/buy/itemX
You can validate that the domain "site.com" is legit.
import java.net.URL; import java.net.MalformedURLException; public class URLValidation{ public static void main(String []args){ try { //URL url = new URL("http://subdomain.site.com.hack.com/site.com"); URL url = new URL("http://subdomain.site.com/pathA/Param1/pathB?paramA=A¶mB=B"); if (url.getHost().endsWith(".site.com") || url.getHost().equals("site.com") ){
System.out.println("OK"); } else { System.out.println("Fail"); } } catch (MalformedURLException e){ } } }
Note: use "endsWith", "contains" allows "site.com.hack.com". Since "getHost()" gets you also subdomains doing a straight comparison forces you to have a list of subdomains, instead of just domains. When using "endsWith" always include a dot "." at the beginning, if not, you would accept "http://hacker-site.com". If you want to accept "http://site.com" you have to add the or statement asking for equals.
Advantage:
1. You only have to make this validation in the redirection controllers, such as login
Disadvantage:
1. You have to keep an updated legit domain list.
2. If the library you used has a flaw you could be vulnerable.
3. If no library is available, making one with regex is extremely prone to error.
4. You have to canonicalize the input and do a validation of the characters, because the attacker might inject a end of line character, causing a header injection or worse, an HTTP Response Splitting.
Potential solutions that won't work or are prone to fail
1. In this situation OWASP first recommendation, avoid redirects, can't be easily achieved.
The second recommendation, don’t involve user parameters, can't be easily achieved neither.
One might think using "back_url=itemX" will solve the problem, the thing is that every section that is not an item that redirects to the login must have a mapping. This will turn out to be a mapping nightmare.
2. Encrypting the URL. Remember that encryption is hard to do it right, and it's targeted at confidentiality. And what you need in this case is integrity and authenticity.
3. Relatives URL won't work in different domains.
4. Create real time mapping server side.
The "favourite" controller generates a Random Token and stores it with the URL in server side (database, memcache, or whatever: it rises the questions of who is the owner, who is responsible for it's up time, monitoring, maintenance, repair)
Keep in mind that they might even be in different data centers. (think of Youtube and Google)
The back_url will be a Random Token like this:
http://site.com/login?back_url=0949jr2oi39493
Another issue with this is keeping all this information quickly accessible, and for a long time.
5. As told before doing your own domain validation is a bad idea, it's very prone to error or you will not accept every type of valid URLs.
6. Fuzzy logic at the front-end.
Scenario A:
1. User goes to see an item2. Presses "buy"
3. A javascript determines if the users is logged or not by watching a Boolean flag cookie. (without HTTPOnly of course)
4A. If the flag says it's logged in, the "buy" actions hits the "buy" controller.
4B. If the flag says it's not logged in, the "buy" actions, performs a POST to the "login" controller, and sends with it, the URL of the visited item, so It knows where to go back.
5B. The login page is displayed, and in a hidden field resides the URL to go back.
6B. If the login is successful it does a redirect to the URL stored in the hidden field, which was send along with the user and password in the POST.
The good thing about this solution is that you don't share any information between "buy" and "login" through the back-end, and you don't store anything in the back-end either.
The first thing you might think is Tampering with the Boolean flag cookie, but if the cookie is forged to look like logged out, and you are logged in, it will send you to the login, nothing wrong with it. If the cookie is forged to look like logged in when you are not, you will go to the "buy" controller, which will tell very quickly that you are not logged in, no harm there.
A potential attacker might create a site that only does a POST to the login, sending the a fake URL to go back, potentially a forged clone of the original login. When the victim does a successful login, it's redirected to the fake login, where it's asked for the user and password again. This way the attacker can exploit this solution.
The big problem is that the login can't be sure who is sending the URL to go back! Using the "Referer" header is not safe as told before, and it's not always there (HTTPS link from one domain to another).
There's probable more bad solutions than good ones, so keep in mind which one you use.
Sunday, March 8, 2015
Port Monitor Tool
Someday I was thinking how can I monitor the status of a big pool of public IPs, and decided to look for something simple that can alert when something changes in an IP, for example a port that was closed is now open.
To be honest I didn't do a thorough google search, but a simple search didn't get me something simple, free and open source, so I decided to make one of my own.
The premises were:
1. If the software suddenly stops, there shouldn't be half a port scan in the database, that would create wrong alerts.
2. If the software suddenly stops, next time it should keep scanning the previous IP and keep the same order.
3. It must be possible to add new IPs, without restarting the software, without the hassle of connecting to the Database.
4. Every scan detail must be stored.
The premises are quite simple.
This script can be used by companies that want to continually audit and monitor they public IPs. They can cross data with tickets or with configuration management.
It can be also be used by consultants, so they can monitor their clients networks. And of course it can be used by attackers or people looking for entry points to look for vulnerabilities for a bug bounty.
I used ruby language, with the library "ruby-nmap" to manage nmap scans.
Every scan has an xml output that is stored in the output folder.
The script is very simple, and probably not very beautiful so it's easy to modify (I believe).
The code is here.
Considerations:
1. It's better to use an external server for the scanning, to be sure that there aren't odd firewall rules that show different results. An example would be an EC2 instance.
2. If you have an IPS and/or a firewall with IPS features, make sure to add the scanner IP in the white list.
3. Nmap is ran with its default Timing settings (-T3), using more aggressive timings can introduce false positives.
4. By default it scans ports: 1-65535.
Limitations:
1. Right now, it just scans 1 IP at a time, so if your pool of IP is very big, it can take a long time to do a full cycle.
2. The tool reports just port state changes, not if the software banner or fingerprint changed, this is in order to simplify and reduce false positives.
3. Output files are stored in the folder "xml", with time this folder can get very big.
4. Log files are in the "log" folder, and they are rotated daily.
How to make it work:
This code was ran with Ruby 2.1.2 at Ubuntu 14.04
The Library for using nmap is:
https://github.com/sophsec/ruby-nmap
These are the required dependencies
After ruby is working, you must create a database with the file database.sql
Before running the script you need to configure:
1. the database name, IP (usually localhost), username, and password. Line
2. e-mail settings, "from@gmail.com" (Line 246).
3. if you want to send e-mails with mandrill you need to get an api-key and replace it where it says 'madrill-api-key' (Line 146)
You will also need to change parameters for e-mail
Someday I'll include a config file, in order to make this easier.
As nmap needs high privileges, you need to run the script with sudo.
sudo ruby port_monitor.rb
Logs are rotated daily, but XML files will pile up with time.
log/activity.log - everything is logged
log/diff.log - just logs port status changes
In hosts.txt you can add new IPs, one IP per line. CIDR notation or any notation is not supported.
After a scan finishes, the script will process the hosts.txt file and rename it hosts.txt.processed.
Let me know if it works, and if there any thing you would change or add.
To be honest I didn't do a thorough google search, but a simple search didn't get me something simple, free and open source, so I decided to make one of my own.
The premises were:
1. If the software suddenly stops, there shouldn't be half a port scan in the database, that would create wrong alerts.
2. If the software suddenly stops, next time it should keep scanning the previous IP and keep the same order.
3. It must be possible to add new IPs, without restarting the software, without the hassle of connecting to the Database.
4. Every scan detail must be stored.
The premises are quite simple.
This script can be used by companies that want to continually audit and monitor they public IPs. They can cross data with tickets or with configuration management.
It can be also be used by consultants, so they can monitor their clients networks. And of course it can be used by attackers or people looking for entry points to look for vulnerabilities for a bug bounty.
I used ruby language, with the library "ruby-nmap" to manage nmap scans.
Every scan has an xml output that is stored in the output folder.
The script is very simple, and probably not very beautiful so it's easy to modify (I believe).
The code is here.
Considerations:
1. It's better to use an external server for the scanning, to be sure that there aren't odd firewall rules that show different results. An example would be an EC2 instance.
2. If you have an IPS and/or a firewall with IPS features, make sure to add the scanner IP in the white list.
3. Nmap is ran with its default Timing settings (-T3), using more aggressive timings can introduce false positives.
4. By default it scans ports: 1-65535.
Limitations:
1. Right now, it just scans 1 IP at a time, so if your pool of IP is very big, it can take a long time to do a full cycle.
2. The tool reports just port state changes, not if the software banner or fingerprint changed, this is in order to simplify and reduce false positives.
3. Output files are stored in the folder "xml", with time this folder can get very big.
4. Log files are in the "log" folder, and they are rotated daily.
How to make it work:
This code was ran with Ruby 2.1.2 at Ubuntu 14.04
The Library for using nmap is:
https://github.com/sophsec/ruby-nmap
These are the required dependencies
sudo apt-get install mysql-server mysql-client sudo apt-get install libmysqlclient-dev sudo apt-get install nmap sudo apt-get install zlib1g-dev gem install rprogram gem install nokogiri gem install ruby-nmap gem install dbi gem install mysql gem install dbd-mysql gem install mandrill-api gem install mail
After ruby is working, you must create a database with the file database.sql
Before running the script you need to configure:
1. the database name, IP (usually localhost), username, and password. Line
2. e-mail settings, "from@gmail.com" (Line 246).
3. if you want to send e-mails with mandrill you need to get an api-key and replace it where it says 'madrill-api-key' (Line 146)
You will also need to change parameters for e-mail
Someday I'll include a config file, in order to make this easier.
As nmap needs high privileges, you need to run the script with sudo.
sudo ruby port_monitor.rb
Logs are rotated daily, but XML files will pile up with time.
log/activity.log - everything is logged
log/diff.log - just logs port status changes
In hosts.txt you can add new IPs, one IP per line. CIDR notation or any notation is not supported.
After a scan finishes, the script will process the hosts.txt file and rename it hosts.txt.processed.
Let me know if it works, and if there any thing you would change or add.
Sunday, February 1, 2015
OpenSSL irks and quirks
Since I believe OpenSSL version 1.0.1d, some odd behaviour was introduced and some stuff just stopped working.
The first thing is that SSL v2 is deprecated, it can't even be forced anymore.
I you need to quickly test if a server supports SSL v2, you are out of luck with new releases of OpenSSL.
Keep in mind that command "ssl_scan" relies on OpenSSL, so that won't work either.
If you must use OpenSSL you can recompile the newer versions to add SSL v2 support.
It's explained in this link.
This deprecation probably affects more security professionals that just want to know if the Sever is properly configured.
What really affects end users, and might bring some headaches to companies is that OpenSSL no longer negotiates SSL v3, or TLS v1.0.
So if your site doesn't support TLS 1.2 (I'm not sure about TLS 1.1), OpenSSL won't establish the connection, as shown below:
And as you guessed already, "curl", the command and implementations for different programming languages relies on OpenSSL.
So "curl" will fail too.
Right now you may be wondering why this is important, since all users access sites through web Browsers, which have no problem negotiating TLSv1.0 or SSLv3, the problem arises if you offer any kind of public API.
3rd party developers will have to sort out this problem, because their software will fail at each API request.
Your company might also offer SDK, so you can place some countermeasures there.
The options are not the best.
This is what curl command offers:
-2, --sslv2 Use SSLv2 (SSL)
-3, --sslv3 Use SSLv3 (SSL)
-1, --tlsv1 Use TLSv1 (SSL)
With new OpenSSL versions SSLv2 is deprecated, because it has long been broken.
When OpenSSL 1.0.1d came out SSLv3 was not vulnerable to POODLE, so many forced it.
The problem is that SSLv3 is on the way to be deprecated, so if you just support TLSv1.0 you have to force that.
For some reason if you force TLSv1.0 with "--tlsv1" or "-1" OpenSSL only tries TLSv1.2, and does not fallback.
I couldn't find an easy way to force TLSv1.0
I did an apt-get upgrade of version Ubuntu 14.04 LTS the 22 of January of 2015.
This is the version I've got of curl and OpenSSL
With this version there seems to be no way to force TLSv1.0
I installed php-curl from apt, and read that since version of 7.34 of curl, this options is supported. (http://curl.haxx.se/libcurl/c/CURLOPT_SSLVERSION.html)
curl_setopt($connect, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_0);
I checked and I had version 7.35
But this is what I got:
This is the PHP script I made to try this:
So in conclusion:
If you don't support yet TLSv1.2 you are out of luck, you are stuck with SSLv3 till its fully deprecated.
The first thing is that SSL v2 is deprecated, it can't even be forced anymore.
I you need to quickly test if a server supports SSL v2, you are out of luck with new releases of OpenSSL.
Keep in mind that command "ssl_scan" relies on OpenSSL, so that won't work either.
If you must use OpenSSL you can recompile the newer versions to add SSL v2 support.
It's explained in this link.
This deprecation probably affects more security professionals that just want to know if the Sever is properly configured.
What really affects end users, and might bring some headaches to companies is that OpenSSL no longer negotiates SSL v3, or TLS v1.0.
So if your site doesn't support TLS 1.2 (I'm not sure about TLS 1.1), OpenSSL won't establish the connection, as shown below:
$ openssl s_client -connect *site that does not support tls1.1 or 1.2*:443 CONNECTED(00000003) write:errno=104 --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 0 bytes and written 305 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE ---
And as you guessed already, "curl", the command and implementations for different programming languages relies on OpenSSL.
So "curl" will fail too.
Right now you may be wondering why this is important, since all users access sites through web Browsers, which have no problem negotiating TLSv1.0 or SSLv3, the problem arises if you offer any kind of public API.
3rd party developers will have to sort out this problem, because their software will fail at each API request.
Your company might also offer SDK, so you can place some countermeasures there.
The options are not the best.
This is what curl command offers:
-2, --sslv2 Use SSLv2 (SSL)
-3, --sslv3 Use SSLv3 (SSL)
-1, --tlsv1 Use TLSv1 (SSL)
With new OpenSSL versions SSLv2 is deprecated, because it has long been broken.
When OpenSSL 1.0.1d came out SSLv3 was not vulnerable to POODLE, so many forced it.
The problem is that SSLv3 is on the way to be deprecated, so if you just support TLSv1.0 you have to force that.
For some reason if you force TLSv1.0 with "--tlsv1" or "-1" OpenSSL only tries TLSv1.2, and does not fallback.
I couldn't find an easy way to force TLSv1.0
I did an apt-get upgrade of version Ubuntu 14.04 LTS the 22 of January of 2015.
This is the version I've got of curl and OpenSSL
$ curl --version curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3
With this version there seems to be no way to force TLSv1.0
I installed php-curl from apt, and read that since version of 7.34 of curl, this options is supported. (http://curl.haxx.se/libcurl/c/CURLOPT_SSLVERSION.html)
curl_setopt($connect, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_0);
I checked and I had version 7.35
$ php -i | grep cURL cURL support => enabled cURL Information => 7.35.0
But this is what I got:
PHP Notice: Use of undefined constant CURL_SSLVERSION_TLSv1_0 - assumed 'CURL_SSLVERSION_TLSv1_0' in /home/ubuntu/testcurl.php on line 13
This is the PHP script I made to try this:
<?php $connect = curl_init("https://yourdomain.com"); # curl_setopt($connect, CURLOPT_SSLVERSION, CURL_SSLVERSION_DEFAULT); # does not connect # curl_setopt($connect, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1_0); # shows error curl_setopt($connect, CURLOPT_SSLVERSION, CURL_SSLVERSION_TLSv1); # does not connect curl_setopt($connect, CURLOPT_RETURNTRANSFER, true); curl_setopt($connect, CURLOPT_CUSTOMREQUEST, "GET"); curl_setopt($connect, CURLOPT_HTTPHEADER, array("Accept: application/json", "Content-Type: text/json")); $api_result = curl_exec($connect); $api_http_code = curl_getinfo($connect, CURLINFO_HTTP_CODE); print $api_result; print $api_http_code; ?>
So in conclusion:
If you don't support yet TLSv1.2 you are out of luck, you are stuck with SSLv3 till its fully deprecated.
Thursday, January 15, 2015
How do I easily create a random Key?
I've met many developers in my life that are conscious enough to use encryption for confidentiality, and are willing to use algorithms such as HMAC for authenticity and integrity when told.
But when they do implement them, they usually stumble when they have to choose a Key.
They usually choose some word (like names) or phrase, which is probably in a dictionary.
This is specially dangerous if the cipher text is being sent to the browser back and forth, or if for example you are trusting HMAC signed requests and parameters.
Some guy might brute force the Key using a dictionary and be able to decipher the data or craft new data, the same with HMAC, he can create valid requests and do harm.
I know that the best approach is to rotate the generated keys, for example every day, and generate the with a Secure Random algorithm.
But not everyone needs that type of security, so if you need to quickly create a Random Key without using web sites that do that for you, this are some commands you can use on Linux or OSx:
If you are using AES-128 you need a 128 bit key or 16 bytes key.
change -c16 to whatever you need.
Since /dev/random generates binary data, its best to transform it to hex string or base64 string.
In the code you should decode the string back to binary data before using it.
For HMAC, the size depends on which Hash algorithm you wanna use. Even if MD5 and SHA-1 are no longer considered secure, their implementation with HMAC has not been proven to be insecure.
But if you can choose, choose the SHA-2 family.
The rule of thumb is to use generate a Key at least the size of the hash output.
For SHA-1 it's 160 bits or 20 bytes:
For SHA-256 as its name tells, 256 bits or 32 bytes.
I don't believe it's necessary to base64 decode this keys when used with HMAC.
If you want to get Hex coded output use xxd as shown below.
But when they do implement them, they usually stumble when they have to choose a Key.
They usually choose some word (like names) or phrase, which is probably in a dictionary.
This is specially dangerous if the cipher text is being sent to the browser back and forth, or if for example you are trusting HMAC signed requests and parameters.
Some guy might brute force the Key using a dictionary and be able to decipher the data or craft new data, the same with HMAC, he can create valid requests and do harm.
I know that the best approach is to rotate the generated keys, for example every day, and generate the with a Secure Random algorithm.
But not everyone needs that type of security, so if you need to quickly create a Random Key without using web sites that do that for you, this are some commands you can use on Linux or OSx:
If you are using AES-128 you need a 128 bit key or 16 bytes key.
$ head -c16 /dev/random | base64
change -c16 to whatever you need.
Since /dev/random generates binary data, its best to transform it to hex string or base64 string.
In the code you should decode the string back to binary data before using it.
For HMAC, the size depends on which Hash algorithm you wanna use. Even if MD5 and SHA-1 are no longer considered secure, their implementation with HMAC has not been proven to be insecure.
But if you can choose, choose the SHA-2 family.
The rule of thumb is to use generate a Key at least the size of the hash output.
For SHA-1 it's 160 bits or 20 bytes:
$ head -c20 /dev/random | base64
For SHA-256 as its name tells, 256 bits or 32 bytes.
$ head -c32 /dev/random | base64
I don't believe it's necessary to base64 decode this keys when used with HMAC.
If you want to get Hex coded output use xxd as shown below.
$ xxd -l 32 -p -c80 /dev/random
Just for your information, /dev/random is a device which generates cryptographically secure random number generator (CSRNG) which are "continuously fed" with with entropy. Use of /dev/urandom is also recommended but not available in all platforms (e.g. OSx).
When you start your computer a pool of entropy starts to generate. Since entropy is hard to get the pool fills slowly, so if you keep getting random numbers from that pool you might deplete it, and you will have to wait for new entropy. If this is your case, because maybe you scripted something that constantly read from /dev/random, you can use the device /dev/urandom, which is a non blocking device. Whenever it's depleted it's starts working a pseudo-random number generator, but when it gets new entropy it uses it.
You can read more about the difference between these 2 devices here.
You can read more about the difference between these 2 devices here.
Tuesday, January 13, 2015
Allow external URLs in my site?
If you are running a website that allows end users to create content, almost any interactive site nowadays, you have to be very careful in how you allow user inputed URLs in it.
For example if you are Facebook you allow users to paste URLs in their wall.
You might even allow URLs, to perform a Javascript or meta tag redirection.
Some developers might opt to use an HTML encoder, breaking many URLs.
More user conscious developers might opt to create very thorough regular expressions allowing most common characters, but that may prevent legit users from inputting real URLs.
So I decided to try to solve this problem, I googled a bit, but couldn't find the answer.
I found some discussions in Stackoverflow, and even some people asked how Stackoverflow actually did it, but no reference to a library or code.
In the past I've used OWASP ESAPI, and I usually recommended it in my classes.
Theres an encoder that I have never used, but thought the name, "encodeForUrl" was self descriptive.
Even the description sounds promising:
"Encode for use in a URL. This method performs URL encoding on the entire string."
So I tried it.
What I got after this sample URL:
"http://www.google.com/dir/?test=test&test2=test<script>alert(1);<ScRiPt>"
Was:
of course the link if used in HTML such as
wont work.
If you look to the source code of the method it's just a call to the URLEncoder.encode method.
It's not a bug, I just misunderstood how that library works.
I could have continued googling or looking at the source code of maybe an open source forum such as phpbb, but I decided to take matters into own hands.
I decided to first make a validation of the URL, being harsh with how it's composed till the domain name finishes. I might left out some legit URLs (not even considered IDN).
if it passes this regex validation
^(https?:\\/\\/)([a-zA-Z0-9-_\\.]+)(:[0-9]{1,5})?((\\?|\\/)(.*))?$
It then URL encodes everything after the domains finishes and decodes certain special characters such as / ? = + & . ,
After writing this post, I started thinking if all of this is needed, why don't I escape the quote and double quote characters, and make sure that HTML attributes are enclosed with them.
At the beginning that seemed to solve the problem in a much easier way.
But I thought of using the newline character %0A in the input (Firefox in OSx doesn't need the %0D for this attack to work)
Some thing like this:
"http://www.victim.com?param=%0A</script><script>alert(1);<script>"
In the response I got something like this:
(the browser does the newline, thats why it's 2 lines here)
Which triggered the "alert(1)".
I didn't test my encodeForUrl with the canonicalized version of
"http://www.victim.com?param=%0A</script><script>alert(1);<script>"
but it shoud look something like this
which is harmless.
(there is no newline as before, it looks like that because of the Post width.)
For example if you are Facebook you allow users to paste URLs in their wall.
You might even allow URLs, to perform a Javascript or meta tag redirection.
Some developers might opt to use an HTML encoder, breaking many URLs.
More user conscious developers might opt to create very thorough regular expressions allowing most common characters, but that may prevent legit users from inputting real URLs.
So I decided to try to solve this problem, I googled a bit, but couldn't find the answer.
I found some discussions in Stackoverflow, and even some people asked how Stackoverflow actually did it, but no reference to a library or code.
In the past I've used OWASP ESAPI, and I usually recommended it in my classes.
Theres an encoder that I have never used, but thought the name, "encodeForUrl" was self descriptive.
Even the description sounds promising:
"Encode for use in a URL. This method performs URL encoding on the entire string."
So I tried it.
What I got after this sample URL:
"http://www.google.com/dir/?test=test&test2=test<script>alert(1);<ScRiPt>"
Was:
http%3A%2F%2Fwww.google.com%2Fdir%2F%3Ftest%3Dtest%26test2%3Dtest%3Cscript%3Ealert%281%29%3B%3CScRiPt%3E
of course the link if used in HTML such as
<a href="http%3A%2F%2Fwww.google.com%2Fdir%2F%3Ftest%3Dtest%26test2%3Dtest%3Cscript%3Ealert%281%29%3B%3CScRiPt%3E">
wont work.
If you look to the source code of the method it's just a call to the URLEncoder.encode method.
It's not a bug, I just misunderstood how that library works.
I could have continued googling or looking at the source code of maybe an open source forum such as phpbb, but I decided to take matters into own hands.
I decided to first make a validation of the URL, being harsh with how it's composed till the domain name finishes. I might left out some legit URLs (not even considered IDN).
if it passes this regex validation
^(https?:\\/\\/)([a-zA-Z0-9-_\\.]+)(:[0-9]{1,5})?((\\?|\\/)(.*))?$
It then URL encodes everything after the domains finishes and decodes certain special characters such as / ? = + & . ,
This double work might not be efficient, but I would rather blacklist everything and then make a whitelist of what I allow, than the other way around, where I cant forget to blacklist some dangerous character.
The code can be found here or below.
You are encouraged to use it and change it, but I'm not responsible for it.
This code assumes that the input is Canonicalized first, so no URL encoded input.
This code assumes that the input is Canonicalized first, so no URL encoded input.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | import java.util.regex.Matcher; import java.util.regex.Pattern; import java.io.UnsupportedEncodingException; import java.net.URLDecoder; import java.net.URLEncoder; public class encodeUrl { public static void main(String[] args) { String testUrl="http://www.google.com/dir/?test=test&test2=test+1,1.<script>alert(1);<ScRiPt>"; if (args.length>0){ testUrl=args[0]; } System.out.println("in: " + testUrl); String pattern = "^(https?:\\/\\/)([a-zA-Z0-9-_\\.]+)(:[0-9]{1,5})?((\\?|\\/)(.*))?$"; Pattern r = Pattern.compile(pattern); Matcher m = r.matcher(testUrl); if (m.matches()){ String out=""; out += (m.group(1)!=null) ? m.group(1): ""; out += (m.group(2)!=null) ? m.group(2): ""; out += (m.group(3)!=null) ? m.group(3): ""; out += (m.group(5)!=null) ? m.group(5): ""; try { out+=m.replaceFirst(URLEncoder.encode(m.group(6), "UTF-8")).replace("%2F","/").replace("%3F","?").replace("%23","#").replace("%3D","=").replace("%26","&").replace("%2B","+").replace("%2C",",").replace("%2E","."); System.out.println("out: " + out); System.out.println("just encode: " + URLEncoder.encode(testUrl, "UTF-8")); } catch (UnsupportedEncodingException e){ System.err.println(e); } } else { System.out.println("out: not a valid url"); } } } |
After writing this post, I started thinking if all of this is needed, why don't I escape the quote and double quote characters, and make sure that HTML attributes are enclosed with them.
At the beginning that seemed to solve the problem in a much easier way.
But I thought of using the newline character %0A in the input (Firefox in OSx doesn't need the %0D for this attack to work)
Some thing like this:
"http://www.victim.com?param=%0A</script><script>alert(1);<script>"
In the response I got something like this:
<a href="http://www.victim.com?param= </script><script>alert(1);<script>" >link</a>
(the browser does the newline, thats why it's 2 lines here)
Which triggered the "alert(1)".
I didn't test my encodeForUrl with the canonicalized version of
"http://www.victim.com?param=%0A</script><script>alert(1);<script>"
but it shoud look something like this
http://www.victim.com?param=%0A%3C/script%3E%3Cscript%3Ealert%281%29%3B%3Cscript%3E
which is harmless.
(there is no newline as before, it looks like that because of the Post width.)
Intro
Hi this is not an introduction of myself, rather its an introduction of this blog.
I'll try to stay true to the title of the blog, writing stuff about building secure code, improving web sites security, and a couple things about breaking them how to prevent it.
I'll also include some side projects regarding security monitoring, or whatever related to security.
What I want to make clear too is that all the code was written by myself during my free time, and that doesn't mean is being used at work. Most of the code are more like PoC than ready to use code.
When I state some opinion, it's my own, and does not reflect my employeers.
Enjoy!
I'll try to stay true to the title of the blog, writing stuff about building secure code, improving web sites security, and a couple things about breaking them how to prevent it.
I'll also include some side projects regarding security monitoring, or whatever related to security.
What I want to make clear too is that all the code was written by myself during my free time, and that doesn't mean is being used at work. Most of the code are more like PoC than ready to use code.
When I state some opinion, it's my own, and does not reflect my employeers.
Enjoy!
Subscribe to:
Posts (Atom)