Using the Zillow Property API to extract Property Listings

Zillow Zestimate

Hey guys, it’s been more than a week since I last posted anything. Today, I am going to show you how to set up the Zillow API to parse details for property and real estate listings. It classifies as a mini Web development project and in general if you wished to extract all the info from Zillow’s Property Listings in one page, well, there you have it!

Alright, so let’s get started.

Step 1: Go to Zillow.com and create a new account by following the steps on the screen. (It is pretty straightforward to sign up here).

Note: You must SELECT the checkbox that says “I am an Industry Professional”. (Unless otherwise specified, normal users are not granted the permission to use the Zillow Web Search APIs).

Just fill your first name and last name, and the website you are using for hosting your web-page for parsing/extracting details from the API. Also, make sure to select the check-boxes labelled Mortgage API, Valuation API and Property Details API. For now, we will mostly be using only the Property Details API.

Step 2: After verifying your e-mail and creating your account successfully, you should have recieved a ZWSID (unique). If not, go to your My Zillow (top right) -> Settings -> Resend my ZWSID.

Note: The ZWSID is pretty much the only authentication you will need/have.

Step 3: Once you have your ZWSID ready, create a new PHP file and insert the following lines into your code:

$zwsid = "insertyourzwsidhere";
$prefixurl = "http://www.zillow.com/webservice/GetDeepSearchResults.htm?zws-id=";
$addr = urlencode("879 W 23rd street");
$cit = urlencode("Los Angeles");
$stat = urlencode("CA");
$csz = $cit."%2C+".$stat;
$query = $prefixurl.$zwsid."&address=".$addr."&citystatezip=".$csz."&rentzestimate=true";
$s = simplexml_load_file(trim($query));
echo json_encode($s);
if($s)
echo "Success" else echo "Failure"

If all goes well, you should have a JSON formatted output as shown :

JSON Sample Output

Step 4: Now, we need to parse each and every detail of the JSON output, doing this is pretty simple as it turns out, but pretty tedious.

First of all, simply do :
echo $query;, this should give you the following link http://www.zillow.com/webservice/GetDeepSearchResults.htm?zws-id=X1-ZWz1dxir1f12bv_6bk45&address=879+W+23rd+Street&citystatezip=Los+Angeles%2C+CA&rentzestimate=true Now you can download the subsequent XML file on that page (pointed to by the link, it will come in handy for parsing).

I will show an example parsed data. The rest can be done in pretty much the same way, for e.g., to extract the Zestimate of the property listing one can simply, traverse through the links of XML name tags upto the leaf element which contains the value for the Zestimate. This is because in the corresponding XML file, the zestimate amount is present inside the zestimate tag which is inside the result tag inside the results tag and which is inside the response tag, here “$s” is made to point to the root XML tag. Also, I have used “money_format()”, a PHP function that displays any number in currency format (by default, in US $).

$zestimate = money_format('%n',floatval($s->response->results->result->zestimate->amount));

Pretty simple stuff, really. You can explore the API’s other functions and calls further on Zillow’s pages for those API’s which I have listed here, and here.

Here is a link  to a working demo version (Keep in mind, this is just a working version and you can beautify the CSS, extract more information, put viewports and make the site into a complete stand-alone App by itself, but that isn’t the purpose of this post). That’s all folks, pretty short and simple post, I will update this blog with better stuff as and when I find time (or learn them (or both)).

Disclaimer: All content is copyright to Zillow (R) and I do not own, market anything on this site. It is simply from an educational and academic perspective.

© Zillow, Inc., 2006-2014. Use is subject to Terms of Use
What’s a Zestimate?

Twitter Simple Tweet Extraction

With data being so important nowadays, it would be nice to collect some data of your own to play around with. I looked around the web and found a simple and easy to use API written in Java to extract tweets and other data from Twitter’s live feed.

Of course there are many other formalities such as Rate Limits on the Tweets, Extracting from Streaming or RESTful sources and many more kinds of OAuth to even be able to post to Twitter if anyone is interested in making an app with Twitter integration. However, recently Twitter released Fabric for this very cause but in this post I will only be using Twitter4j. [1]

So without further ado, the following is a simple way to extract Tweets from Twitter.

Step 1:

Install Java and Eclipse (any version) before proceeding to step 2. For help on either, check the references below [2].

Step 2:

In Eclipse, create a New Project. Call it TwitterExtractor or something similar. (Do File -> New -> Java Project). Create a new Class File called TweetExtractor or something similar. (Again, either create a new class file by doing File -> New -> Class or Right Click on the Project in the package explorer and New -> Class).

Step 3:

Before proceeding to the actual code there are a few things that must be configured to extract data from Twitter. First, go to Twitter’s Developer Website. Select the “Create a New App” button and in the subsequent page, fill in the details for the Application Name, Description and Website (if you don’t know what to put try http://www.google.com). Agree to the Terms and Conditions and click on Create Application.

Step 4:

In the Application homepage, you should have a screen as similar to below.

The circled area indicates the Keys and Access Tokens tab. Before going there, create a new File called twitter4j.properties (case sensitive) and insert the following 5 lines there.

debug=true
oauth.consumerKey=*********************
oauth.consumerSecret=******************************************
oauth.accessToken=**************************************************
oauth.accessTokenSecret=******************************************


After this step, we return to the Twitter App home page. Substitute the starred out characters with actual keys and access tokens. To acquire this, go to the circled tab. The first two should be readily available on the page. Navigate to the bottom of the page and click on “Generate my Access Token”. In moments, you will receive the final two properties to set as well, namely access Token and accessTokenSecret. Now you have configured your App to make API calls (by default these calls are read-only, you can change them in App Permissions on the same page).

Step 5:

Now that you have configured the API. Go to Twitter4j’s website and download the Twitter4j.zip file. Extract the zip file.

In Eclipse, select the Properties option from the Project drop down list. (Alternately, you can Right click on the project in the Package Explorer pane, then do Build Path -> Configure Build Path). The following window should appear. Select the Add External JARs option.

Navigate to the location where you extracted the twitter4j.zip file. Go to the lib folder and then to the file named “twitter4j-core-x.y.z.jar” (x.y.z = 4.0.2 is the version I am using, but older versions should work fine too). Select the JAR file and click on OK to add the external JAR file to the project. (Note: if you have done this step correctly there should be no errors when copying the code below.)

Step 6:

Return to the Java Project and in the Class file, add the following code:


import twitter4j.*;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.List;
public class TweetExtractor {
public static void main(String[] args)
{
Twitter twitter = new TwitterFactory().getInstance();
try {
Query query = new Query("");
QueryResult result;
File file = new File("outputs/tweets.txt");
FileWriter fw = new FileWriter(file, true);
BufferedWriter bw = new BufferedWriter(fw);
do {
result = twitter.search(query);
List<Status> tweets = result.getTweets();
for (Status tweet : tweets) {
System.out.println("@" + tweet.getUser().getScreenName() + " - " + tweet.getText() + "\n");
bw.write("@" + tweet.getUser().getScreenName() + " - " + tweet.getText() +"\n");
}
}
while ((query = result.nextQuery()) != null);
System.exit(0);
bw.close();
} catch (TwitterException te) {
te.printStackTrace();
System.out.println("Failed to search tweets: " + te.getMessage());
System.exit(-1);
} catch (IOException e) {    e.printStackTrace();    }
}
}

In Eclipse, create a New Folder called “outputs” before you run the above code.

Also, the line that says Query query = new Query(""); requires actual arguments. For e.g, to search for Tweets related to Cricket, do Query query = new Query("cricket");

And that’s it! Just 6 steps for extracting your very own data. If you would like to view your Tweets, go to the outputs folder and open up tweets.txt.

A word of caution: Rate Limits are set by default to 180 tweets per 15 minutes (for most users this is the case, due to security reasons). For more info on how to extract more tweets and continuously without the need to call the program every time. However, this is just a basic application just meant to give a kind of start to collect data and to play around with it. [3]

P.S: It seems in certain Eclipse versions the Java is below JDK 1.5, Eclipse might throw an error for the List <Status> for parametrized lists but this is fine, Eclipse can fix this on its own and even otherwise you can use some other kind of data structure to store the tweets, users etc.

References :

[1] Twitter4j – A twitter API for Java.

[2] Installing Eclipse – an IDE for Java and Android projects

[3] Rate Limits and what not – for Twitter Developers