OWASP ZAP Reconnaissance – Without Permission!

How To Assess a Third Party Web Site or Cloud Service with the OWASP ZAP Attack Proxy When You Don't Have Permission to Pentest

As a security professional, you will often be asked to give your opinion or assessment on the security of a third-party web site or cloud service. The person asking the question will usually have no authority to give you permission to run a penetration test on the remote site, and the chances that you can secure permission from the remote site's owner will also be remote.

If this happens to you, are you stuck? Actually, the answer is no. There is plenty of reconnaissance you can perform on a third-party service without requesting special permission, as long as you have a solid attack proxy and a plan.

11 courses, 8+ hours of training

Learn cybersecurity from Ted Harrington, the #1 best-selling author of "Hackable: How to Do Application Security Right."

See Courses

Introducing the OWASP ZAP Attack Proxy

One of the top free tools in application security and pentester toolboxes these days is the OWASP ZAP attack proxy. ("ZAP" stands for "Zed Attack Proxy.") In pentest mode, this tool can map a site, attempt exploits and fuzz input, but using these capabilities against a third-party site without the owner's permission can be an invitation for trouble.

Fortunately, third-party site owners usually grant permission during trial and demo periods to try out their site and service using normal web browsers and mobile devices. The trick to good (and legal) reconnaissance is to only capture and analyze the traffic available in the trial period, and this happens to be another application at which OWASP ZAP excels.

Setting Expectations and Getting Internal Permission

Before you do anything technical, however, you should set expectations and get some cover from the person who requested the assessment. In the process of doing this, you will also lay out your plan, request access to a demo account, and explicitly tell the requester what you are not going to do (i.e., "hack" or run a "pentest"). After you do more of these evaluations you will surely develop your own template, but the following message will get you started:

Dear [Requester],

I promise to complete my assessment of [Service_To_Analyze] and provide a recommendation as to whether or not it's safe enough to use for [Purpose_of_Service] by [Reasonable_Time]. To get started, I need you to provide me with a trial or demo account by [Soon].

Since we do not have permission from the remote site to perform a penetration test, my evaluation will use a non-invasive analysis of the traffic already available to us as demo or trial users. (i.e., We are not "hacking" their site; we are just looking very carefully at the information they share with everyone else who evaluates their service.)

Is this OK?

Regards,

[Your_Name], [Your_Title]

For your own protection, please do NOT begin your research until you get:

1. A positive acknowledgment to your "Is this OK?" question in writing.
2. A working demo account, which you have tested using a regular Web browser.

Planning Your Reconnaissance

If you don't get to hack, what can you do? Actually, the list of items you can investigate from normal traffic is often long enough to make a judgment call on the security posture of the target service. In this article we'll cover "just" nine, but you could certainly look at many more.

1. Use of HTTPS to protect traffic
2. Quality of SSL certificate
3. Avoids client-side secrets or authentication
4. Up-to-date software
5. Secure site headers
6. Proper location and protection of vital assets
7. Avoids information leakage through "extra" fields
8. Access controls on web APIs (sometimes)
9. Evaluation of legal and privacy policy

Preparing OWASP ZAP and Your Browser

To use OWASP ZAP in a noninvasive, passthrough mode, you need to set ZAP up as a proxy.

1. From ZAP's main menu, select "Tools | Options".
2. In the "Local Proxy" section, set the address and port your browser will use (The defaults are an address of "localhost" and a port "8080").
   
3. In the "Dynamic SSL" section, click the "Generate" button to create a CA certificate to use to facilitate your HTTPS connections.
4. Still in the "Dynamic SSL" section, click the "Save" button to save a copy of this CA certificate as a *.cer file (You will want to import this CA into your browser soon to avoid "untrusted SSL certificate" blocks).
5. Pick the Web browser you want to use to examine the remote site. Since I use Chrome for my daily browsing, I usually use Firefox as my secondary browser for Web analysis.
6. Open your selected Web browser and set up its proxy settings, located here in current versions of Firefox ("Options" dialog, "Network" tab, "Connection" section, "Settings" button):
   

   
   

   ...IE ("Settings" icon, "Internet Options" option, "Connections" tab, "LAN Settings" button)
   

   
   

   ...and Chrome ("Settings" dialog, "Slow advanced settings…" link, "Network" section, "Change proxy settings…" button, then see "IE" entry above because it uses the same "Internet Properties" dialog as IE).
7. Once you have configured your proxy, you should also import the proxy's SSL CA certificate so your browser will not warn you about a bad certificate (the proxy's certificate) every time you contact an HTTPS server. In Firefox, the list of trusted CA certificates is available from the "Certificates" tab in the program options. In IE and Chrome, the list of trusted CA certificates is also available through each browser's options, but is actually saved in the local Windows operating system, not the browser itself.
8. To test all this, restart your selected browser, make sure OWASP ZAP is running, and contact an HTTPS-protected site like https://www.google.com. You may immediately see a certificate warning page like this:
   

   
   

   ...because the certificate presented by the proxy does not match the target, but you should have an option to "Add Exception" or "Proceed" because you already added the proxy's certificate to your list of trusted CAs (E.g., you get a yellow warning you can ignore instead of a red error that stops you from proceeding).

To proceed, click the available link and inspect the provided certificate. If you performed your steps correctly you will "OWASP" all over it.

Click the "Permanently store this exception" box and then click the "Confirm Security Exception" button (or similar controls) to dismiss the error and proceed with your connections.

Bypassing the Proxy for Certain Sites

As you perform research on various sites, you may also want to set up a list of sites that will not be queried through the proxy. These settings are usually near your browser's other proxy settings. For example, in Firefox, they are immediately below the proxy host and port settings.

Performing Your Reconnaissance

Now that you have OWASP ZAP and your browser set up, let's proceed with some reconnaissance. Our sample target today will be a web services company called EventMobi, which hosts a public demo at http://eventmobi.com/mfg2015/attendees/76204. (Remember, we're being non-invasive, so public resources are best!)

Use of HTTPS to protect traffic

To get started, simply open up the public demo link in your browser. (Do NOT enter it into the inviting little "URL to attack" field in OWASP ZAP!) Since we're going through a proxy that may tie things up, you may need to refresh your browser (or perform the "Resolving Missing Images" procedure below) a few times to get all the content you want, but in less than a minute you should see a valid web page in your browser:

...and some folders will start to appear in ZAP under the "Sites" list.

Notice that pages are already listed as coming from "http" or "https." In the case of our sample site, it's clear that all content is being served from http, and that we haven't been pushed to https automatically.

To research this further and see if https is an option if we want secure transport, we can go back to the browser and simply change the main URL from http://eventmobi.com/mfg2015/attendees/76204 to https://eventmobi.com/mfg2015/attendees/76204. In this case, the site doesn't load (there is an eternally spinning "loading" circle), so we switch to a "normal" browser (not going through the proxy) to confirm the lack of HTTP support. Since the behavior is the same with our proxied and normal browsers, we reach an unfortunate conclusion: this site doesn't use HTTPS by default, and won't support HTTPS if we ask for it.

Resolving Missing Images

To resolve images and other resources that come from other sites, you may need to perform the following procedure, especially if they are being served from HTTPS resources.

1. Right-click the non-resolving resource (such as an image) and right-click "Copy Image Location" (or similar).
2. In a new tab, paste the URL from the previous step.
3. Click through any certificate resolution or other site-specific errors until the specific resource resolves.
4. Go back to the tab with the main application and refresh it to resolve all resources from the same site.

Assessment of Sample Site

For "use of HTTPS to protect traffic," we conclude that this site is weak because it doesn't use HTTPS by default, and won't support HTTPS if we ask for it.

Quality of SSL certificate

While we're on the subject of HTTPS, we should step out out of our proxied browser again and inspect the real X.509 certificate being offered up by our target site. We can usually do this by clicking the lock or certificate icon near our page URL. Here is what that looks like in Chrome (my daily, non-proxy browser).

To get even more information, click "Certificate information." There are generally three things you want to look for here.

First, make sure that there is a fully trusted path to a legitimate CA. This certificate is in good shape.

Next, check the "CN" on the certificate (usually in the "Subject" field). In this case, the certificate looks like a "wildcard" certificate because the CN is for the entire domain rather than a specific server. This is often a good sign, because wildcard certificates indicate that a company is large or stable enough to spend the extra money on a site-wide certificate.

Finally, take a look through the other fields on the certificate. In this case, there is an interesting list of what appears to be customer-specific sites in the "Subject Alternative Field." If you plan on using your own domain name through the provider, having this list broadcast to anyone who connected to the provider's Web site might or might not make you nervous.

Assessment of Sample Site

For "quality of SSL certificate" we conclude that this site is OK because it uses a valid X.509 certificate, but could be better since the certificate seems to be leaking the URLs of some other customer sites.

Avoids client-side secrets or authentication

Now that we're done with SSL and certificates, let's look at the application itself. To see how it works, click throughout the application in your proxied browser. (In other words, try to go everywhere you can through links, buttons and forms.)

Once you have a nice set of data, return to the ZAP console and open the tree that corresponds to our target site (in this case, "http://evenmobi.com"). Then drill down into a particular request and click the "Response" tab to see what the target site is telling us.

What we are mostly looking for here is JavaScript or other client code that is performing authentication requests, especially passwords accidentally sent to the client. (To help detangle hard-to-read JavaScript, remember to use a JavaScript Beautifier.) Our target site has a lot of its resources in a "webapp/view/high/js" tree, so we can drill down there to look at individual files.

Another place to look for potential exposure is the history of requests at the bottom of the ZAP console. In this case, ZAP has highlighted a "*.js" file that contains the word "password" and warrants further inspection.

A complete analysis of client code, authentication routines and unsafe password handling is beyond this article (and could take more time than the rest of the steps combined), but suffice it to say that our target site passed its inspection.

Assessment of Sample Site

For "avoids client-side secrets or authentication" we conclude that this site is fine because it doesn't do client-side authentication and keeps its passwords safe on the server.

Up-to-date software

Most Web sites depend on a variety of third-party libraries and applications, and many sites lag behind the most current and secure versions of these components. This problem is so widespread that OWASP continues to track it as #9 in its Top Ten Web Vulnerabilities.

Fortunately ZAP can help us look for these in three ways.

First, we can look at Web site headers for Web software version numbers. Unless it's suppressed by the target server or intervening firewall, this information will be displayed in any content response from the site.

In this case we see that the site is claiming to run version 1.1.19 of the nginx Web server. Checking the nginx release history, we see that that version was released in April 2012 - almost three years ago. We can also look up nginx security vulnerabilities to see that our target server may have a number of "medium" severity vulnerabilities (fortunately no "major" vulnerabilities), including:

• SSL session reuse vulnerability
• Request line parsing vulnerability
• Memory disclosure with specially crafted HTTP backend responses
• Vulnerabilities with Windows directory aliases

Second, we can use ZAP to look for application environment versions. These may be displayed if we access "Web application" files like *.php, *.aspx, etc. (Hiding this information is a best practice, so it may not appear on all sites.)

In this case we see that the remote site is running PHP version 5.3.10, and is also probably running Ubuntu with a Linux kernel version of 3.15. That suggests that the operating system was updated at least once in the past year (OK), but that the PHP environment has been deprecated (all support, including security support, terminated in mid-2014).

Finally, we can also use ZAP to check standardized JavaScript libraries and other includes for vulnerable versions. These may be easy to find in the list of downloaded assets:

...or may be buried in inline page includes (which requires looking through downloaded content).

In this case, we can check at least three standard libraries for known vulnerabilities:

• socket.io (version 0.9.11) - current version is 1.3.3; this version appears safe
• jquery (version 1.7) - current version is 1.9; this version appears safe
• keen (version 2.1.0) - current version is 3.2.2; this version appears safe
• etc…

Assessment of Sample Site

For "up-to-date software" we conclude that this site is worrisome because it is running a pretty old version of its Web server software with several known vulnerabilities and an out-of-support version of PHP. (The application JavaScript components appear fine, but the Web server and PHP issues are enough to cause concern.)

Secure site headers

A sign that a service takes security seriously is the use of special Web site headers designed to prevent XSS and related exploits. Many of these headers are detailed on other sites, but some of the best ones to look for are:

• X-Frame-Options: SAMEORIGIN or X-Frame-Options: DENY
• X-Content-Type-Options: nosniff
• (Missing) X-Powered-By…
• Strict-Transport-Security
• Content-Security-Policy

Using ZAP, these would show up in the Response section of most file requests. Unfortunately, none of the secure headers we would like to see are used by our target site, and as we saw above, the target site quite happily uses an "X-Powered-By" header to tell us about the underlying application (PHP) and OS (Ubuntu Linux) environment.

Assessment of Sample Site

Our target site does not use any "secure site headers," and does not suppress headers that may provide helpful information to hackers.

Proper location and protection of vital assets

One of the many things ZAP does well is organize detected assets by site of origin. This makes it easy to see where a target site stores its assets and runs its application. In the case of our target site we can see resources come from newrelic.com, linkedin.com, amazonaws.com and google-analytics.com, among other places.

For now we will ignore the tracking and advertising sites and zero in on resources pulled from Amazon's S3 storage service. To see these, open up the related tree until you can drill down to an asset.

Notice that the full URL of each asset is available in ZAP. To see whether or not an asset is accessible without any access control, copy its URL into an "incognito" browser window in a non-proxy browser. (ZAP provides a right-click "Copy URLs to Clipboard" option for this exact purpose.)

If you can pull up the resource in a fresh, incognito browser session, there is generally no access protection and anyone with the URL to the resource can access it. In some cases this is OK, but if confidential information is ever accessible in this way you could have a leak on your hands.

Assessment of Sample Site

In terms of "proper location and protection of vital assets" there are a number of company-specific assets that are stored on publicly-accessible storage at Amazon. Since all the information we plan to store in this application is public anyway, this is OK, but we would need to see a different storage mechanism in place with enforced access controls if we planned to use the service with any confidential information.

Avoids information leakage through "extra" fields

It is common for Web applications, particularly Web services, to provide "extra" information with requests that client-side (usually JavaScript) code will filter out. However, all this information is easily accessible to interested parties, including you.

To look for this type of information using ZAP, look for large page or Web service requests in your history, then dig into the fields provided. Remember to cut/paste data in JSON prettifiers and other tools if you need help visualizing it.

For example, here is an abbreviated response from our sample site:

[plain]

{

"response":{

"id":"76204",

"name":"Attendees",

"items":[

{

"id":"1812359",

"first_name":"James Avery",

"about":"",

"image50":"50mfg-m-01.jpg",

"image100":"100mfg-m-01.jpg",

"title":"Support Specialist",

"company_name":"Metropolitan Financial Group",

"website":"http://www.eventmobi.com",

"facebook":"http://www.facebook.com/eventmobi.com",

"twitter":"http://twitter.com/eventmobi",

"linkedin":"http://www.linkedin.com/in/eventmobi",

"url":"http://api.eventmobi.com/en/api/v1/events/MFG2015/sections/76204/items/1812359"

},

{

"id":"1812360",

"first_name":"Joe Baker",

"about":"",

"image50":"50crop_547c946dd931f_128_(1).jpg",

"image100":"100crop_547c946dd931f_128_(1).jpg",

"title":"CEO",

"company_name":"Metropolitan Financial Group",

"website":"http://www.eventmobi.com",

"facebook":"http://www.facebook.com/eventmobi.com",

"twitter":"http://twitter.com/eventmobi",

"linkedin":"http://www.linkedin.com/in/eventmobi",

"url":"http://api.eventmobi.com/en/api/v1/events/MFG2015/sections/76204/items/1812360"

},

[/plain]

This is a block of JSON that shows that the target site will happily dump a complete attendee list, including self-provided contact information.

Assessment of Sample Site

In terms of "Avoids information leakage through "extra" fields" the target site is OK. Although interesting fields such as "linkedin" are revealed, they must be added by individual users and are probably easy to find through other publicly available sources.

Access controls on web APIs

After you have identified a few key API service calls in ZAP, copy the URLs out and try them in your incognito Web browser to see if they are protected by any access controls.

In the case of our sample site, there do not appear to be any access controls on the API.

Assessment of Sample Site

In terms of "access controls on Web APIs" there do not appear to be any access controls. This may be OK for public information, but if information such as lists of attendees is sensitive, then we may be concerned.

Evaluation of legal and privacy policy

Finally, you can put ZAP down for a minute and read some legalese. Go back to the target site using a non-proxy Web browser and find its legal terms, privacy or security policy. For our target site, a privacy policy can be found at http://www.eventmobi.com/legal/privacy-policy/

At a high level, you are looking for promises of privacy or security made in the legal policy that do not seem to be backed up with technical controls.

As it turns out, our target's privacy policy is weak, but accurately describes what they do or do not do.

In a section about the type of information collected and its use, our target accurately lists what they collect and states that the information of participants is "publicly available online" (as we saw while looking at Web service access controls).

In the "Security" section we see a statement about "suitable procedures" to protect information collected online. This is vague enough to encompass what we saw in our investigation (e.g., no encryption for public information), but without more inspection we have to take our target on its word.

Finally, another "Security" section contains a vague promise of "security measures," and then honestly puts the onus of not posting anything confidential on the users.

Assessment of Sample Site

Our "evaluation of legal and privacy policy" did not give us great confidence in the security of the site, but it agrees with our overall technical assessment. For that reason, it strikes us as an honest policy.

Filing Your Report

Now that you have completed your analysis, it's time to provide a recommendation based on your limited information. Remember that your requester is primarily looking for a simple answer to this question: "Is this service safe enough for the job?" A typical results report might resemble the following template.

Dear [Requester],

On [Date_Of_Request] you asked me to perform a non-invasive ("zero hacking") assessment of [Service_To_Analyze], and I used the trial credentials you shared with me on [Date_Credentials_Provided] to perform this assessment.

In my professional security opinion, [Service_To_Analyze] appears to be [fit/unfit] to use for [Purpose_of_Service] because [Heart_Of_Your_Recommendation].

Please understand that my recommendation is based on limited information, but I was able to evaluate the following factors in my assessment. [(Complete assessment details are attached below.)]

[PRO/CON/NA]: Use of HTTPS to protect traffic

[PRO/CON/NA]: Quality of SSL certificate

[PRO/CON/NA]: Avoids client-side secrets or authentication

[PRO/CON/NA]: Up-to-date software

[PRO/CON/NA]: Secure site headers

[PRO/CON/NA]: Proper location and protection of vital assets

[PRO/CON/NA]: Avoids information leakage through "extra" fields

[PRO/CON/NA]: Access controls on web APIs

[PRO/CON/NA]: Evaluation of legal and privacy policy

[Risk_Acceptance_Statement]

Regards,

[Your_Name], [Your_Title]

[Optional: cut/paste or attach the rest of your analysis.]

For example, an evaluation of our sample target might yield the following report.

Dear Mr. Lumbergh,

On February 28, 2015 you asked me to perform a non-invasive ("zero hacking") assessment of EventMobi (http://eventmobi.com) for the upcoming Initech convention, and I used the trial credentials (http://eventmobi.com/mfg2015/attendees/76204) you shared with me on March 2 to perform this assessment.

In my professional security opinion, EventMobi appears to be fit to use for our completely public event program because the service appears to be designed to serve public information to the public. However, I would not be able to recommend it to store or serve up any confidential information because sensitive data controls do not seem to be available, and outdated software appears to be in use. We should also tread with caution if we believe our attendee list is sensitive information.

Please understand that my recommendation is based on limited information, but I was able to evaluate the following factors in my assessment.

CON but N/A for our use: Use of HTTPS to protect traffic. (Doesn't use HTTPS by default, and won't support HTTPS if we ask for it, but we don't need HTTPS)

PRO but N/A for our use: Quality of SSL certificate. (Uses a valid X.509 certificate, but the certificate may be leaking other customer site names).

PRO: Avoids client-side secrets or authentication

CON: Does not use up-to-date software (Uses 3-year-old web server version with known vulnerabilities and out-of-support version of PHP.)

CON: Does not use secure site headers

CON but N/A for our use: Uploaded company assets are publically available on Amazon storage. (This is OK for our current application since we are only publishing public information, but would not be acceptable if confidential information was involved.

PRO: Avoids information leakage through "extra" fields

CON but N/A for our use: Does not use access controls on web APIs (this exposes a complete list of registered attendees to the public, but since this information is easily available anyway to the public - who can register with a fake account - this poses little additional risk)

PRO: Evaluation of legal and privacy policy (weak but honest)

Please note that use of this application requires us to accept the two remaining risks identified above (i.e., does not use up-to-date software or secure site headers).

11 courses, 8+ hours of training

Learn cybersecurity from Ted Harrington, the #1 best-selling author of "Hackable: How to Do Application Security Right."

See Courses

Regards,

Jonathan Lampe, CISSP