I recently started a personal project to monitor the real estate market in Houston to gain insight into trends in pricing, neighborhood composition, construction, etc. One vital piece of the puzzle was reliably and efficiently gathering detailed information on specific properties to monitor over time. Using Zillow.com’s Developer APIs and the HTML Agility Pack, I was able to spin up a prototype over coffee one Sunday morning, and I wanted to share my experience using this powerful C# library.

The Zillow API

The majority of modern websites use JSON as the data exchange format, as it is easily consumed by the widely-used JavaScript language, but there are a significant number of sites (including Zillow.com) that use XML as the exchange format. There are advantages and disadvantages to each, but both formats are straightforward to parse and are human-readable. 

Zillow provides several useful APIs, and I chose GetDeepSearchResults because it encapsulates most of the data I want to track in my application. The documentation is refreshingly complete and useful: you can see details about each field, the XML schema, error codes, and an example usage. After obtaining my developer API key, I scratched out a few lines of code to load the results.

 private void LoadResults(string property)
 {
     // Argument "property" is a URL-encoded string containing a single property address, city, and state
     string addr = "http://www.zillow.com/webservice/GetDeepSearchResults.htm";
     string key = "?[My_API_Key]&";
     string results = String.Empty;

     // Create a new HTTP client to communicate with the Zillow server
     HttpClient client = new HttpClient();
     client.BaseAddress = new Uri(addr);

     // Load the results of the querty
     HttpResponseMessage response = client.GetAsync(key + property).Result;
     if (response.IsSuccessStatusCode)
     {
         results = response.Content.ReadAsStringAsync();
     }
     else
     {
         Debug.WriteLine("Web call failed with code " + response.StatusCode);
     }
     return results;
 }

 private void LoadResults(string property)
 {
     // Argument "property" is a URL-encoded string containing a single property address, city, and state
     string addr = "http://www.zillow.com/webservice/GetDeepSearchResults.htm";
     string key = "?[My_API_Key]&";
     string results = String.Empty;

     // Create a new HTTP client to communicate with the Zillow server
     HttpClient client = new HttpClient();
     client.BaseAddress = new Uri(addr);

     // Load the results of the querty
     HttpResponseMessage response = client.GetAsync(key + property).Result;
     if (response.IsSuccessStatusCode)
     {
         results = response.Content.ReadAsStringAsync();
     }
     else
     {
         Debug.WriteLine("Web call failed with code " + response.StatusCode);
     }
     return results;
 }

The LoadResults function takes in a well-formed URL-encoded property listing string that meets the requirements listed in the API documentation, and returns either an empty string (if the API call failed) or an XML-encoded string containing the property details.

Parsing with Agility

I could have spent some time brewing up a RegEx-powered parser for the XML results – after all, this is a well-documented API, and I know what to expect from each call. But why re-invent the wheel? It’s also worth pointing out that not all APIs (or web pages, which you can also easily scrape with the HTML Agility Pack) are as consistent and well-formed, so your homebrewed parser might work for a while until it tries to handle a malformed response. 

Using the Agility Pack and the Zillow documentation, I can easily parse out the XML tags I care about without writing a single regex.

 private PropertyData ParseResults(string results)
 {
     // Create an instance of the PropertyData to store the results
     PropertyData data = new PropertyData();

     // Load the API result string
     HtmlDocument doc = new HtmlDocument();
     doc.LoadHtml(results);

     // Find the parameters that will be tracked
     var zpid = doc.DocumentNode.Descendants("zpid").FirstOrDefault().InnerText;
     var finSize = doc.DocumentNode.Descendants("finishedSqFt").FirstOrDefault().InnerText;
     var zestValue = doc.DocumentNode.Descendants("amount").FirstOrDefault().InnerText;
     var baths = doc.DocumentNode.Descendants("bathrooms").FirstOrDefault().InnerText;
     var beds = doc.DocumentNode.Descendants("bedrooms").FirstOrDefault().InnerText;

     // Parse these value strings into their expected data types
     data.Zpid = int.Parse(zpid);
     data.FinishedSize = int.Parse(finSize);
     data.ZestValue = double.Parse(zestValue);
     data.Bathrooms = double.Parse(baths);
     data.Bedrooms = double.Parse(beds);

     return data;
 }

 private PropertyData ParseResults(string results)
 {
     // Create an instance of the PropertyData to store the results
     PropertyData data = new PropertyData();

     // Load the API result string
     HtmlDocument doc = new HtmlDocument();
     doc.LoadHtml(results);

     // Find the parameters that will be tracked
     var zpid = doc.DocumentNode.Descendants("zpid").FirstOrDefault().InnerText;
     var finSize = doc.DocumentNode.Descendants("finishedSqFt").FirstOrDefault().InnerText;
     var zestValue = doc.DocumentNode.Descendants("amount").FirstOrDefault().InnerText;
     var baths = doc.DocumentNode.Descendants("bathrooms").FirstOrDefault().InnerText;
     var beds = doc.DocumentNode.Descendants("bedrooms").FirstOrDefault().InnerText;

     // Parse these value strings into their expected data types
     data.Zpid = int.Parse(zpid);
     data.FinishedSize = int.Parse(finSize);
     data.ZestValue = double.Parse(zestValue);
     data.Bathrooms = double.Parse(baths);
     data.Bedrooms = double.Parse(beds);

     return data;
 }

That’s all there is to it! No bulky regular expressions or clunky homebrewed parsing code. For clarity, I stripped out a lot of input sanitation and error handling code, so don’t forget to check for exception conditions!

Other Applications

You may have an inkling that a parsing tool like the HTML Agility Pack can work for more than just parsing XML responses. You would be correct. Another key area where this library can come into play is data scraping (or web data extraction). If an API is not available or doesn’t give you all of the information you need, you can load the web page and parse the whole thing for the data you need. 

It’s very important to note that web scraping may violate the Terms of Use for a site, and if your scraping tool misbehaves and loads the webpage at a high rate, you run the risk of compromising the site’s resources and awaking the angry IT department kraken! Please be considerate, review the legality of your application, and scrape responsibly!

If you’ve used the HTML Agility Pack in an application or have questions about how I used it in mine, leave a comment below! Happy coding.

Learn more about DMC’s web application development.

The post Easy XML Parsing in C#/.NET With the HTML Agility Pack appeared first on DMC, Inc..

In this previous blog post, I explained how to integrate Microsoft Dynamics CRM with SharePoint 2010 using an External Content Type (ECT) linked to the Dynamics SQL Server instance. Today I will explain how to integrate Microsoft Dynamics CRM Online with SharePoint 2010, also using an ECT. Since CRM Online data is only accessible via web services, we will use a WCF web service-based ECT.

First a note about why CRM Online data is only accessible via web services. CRM Online is Microsoft’s cloud-hosted CRM solution, so the location of the databases containing your data is not local. For this reason, we will not be able to pursue the strategy outlined in the first blog post, but will instead use the web services provided by CRM Online.

Next, I will note that the CRM Online web services can only be used by a client running .NET 4. SharePoint 2010 is built with .NET 3.5, so SharePoint’s Business Connectivity Services (BCS) cannot connect directly to the CRM Online web service. Instead, we will build an intermediary web service that runs on an application pool using the .NET 4 runtime and has a BCS-compatible interface.

To start, set up a CRM Online account (if you do not already have one )and associate it with a Windows Live ID. Once you can access your CRM Online account in a web browser, download the CRM 2011 SDK.

Within the “bin” directory of the CRM SDK find the crmsvcutil executable. This tool will generate the CRM data model used by your CRM Online site in code so that we will be able to use Linq to query CRM data. The syntax of the tool follows.

crmsvcutil.exe /url:https://[Site/ Name].crm.dynamics.com/XRMServices/2011/Organization.svc /out:CrmDataModel.cs /username:”[Live ID]” /password:”[Password]”

The file CrmDataModel.cs will be used in your web service.

Next, open up Visual Studio 2010 and create a new project based on a WCF Web Service template. Be sure to set the .NET Framework version to 4. Add CrmDataModel.cs to the project.

I would also suggest deleting the default web service item from the project and adding a new one with a more specific name.

Next, flesh out the web service so that it can gather the desired entities from CRM Online and deliver them to the web service consumer. Please see this blog for example code on how to use the CRM data model. If designing the web service to be used as part of an ECT, keep in mind the CRRUD operations (Create, Read List, Read Item, Update, Delete) that an ECT supports.

Next test out the web service. Visual Studio 2010 has a great point-and-click client for testing out web services that is opened when you begin debugging the web service through the IDE. If testing on a server that does not have Visual Studio 2010 installed, I have found that WCFStorm Lite is a great tool for reproducing the behavior of the Visual Studio WCF client.

Once you have verified that the web service is functional, you will have to create a web site and application pool to host it. Please remember that since this web service is unsecured, it should run either on a server that is not externally accessible or on a closed port. When adding the web site, make sure to grant the user IIS_IUSRS read/execute permissions to the site. This can be done through IIS Manager by right-clicking on the site folder and selecting Permissions. Also, remember when creating the application pool to be used by this site to set it to use the .NET 4 Framework. This can be done by selecting the application pool in IIS Manager and configuring Basic Settings. Lastly in IIS set the “Load User Profile” option to true under the Advanced Settings of the application pool.

Add the web service to the site by selecting the WCF project folder you created as a web application. This should be the location of the .svc files generated when building the project. You may need to grant permissions to this application to IIS_IUSRS. Fire up a browser and navigate to the new site you created – you should receive a page summarizing the web service.

If installing the web service on a server different than that hosting Visual Studio, be sure to copy the .svc files, the web.config configuration file, and the code assembly containing the service’s logic. This assembly should be placed in a “bin” folder that resides at the same location as the web application folder holding the .svc files.

To test the web service, debug one of the service’s code-behind files, or start up WCFStorm. If debugging through Visual Studio 2010, the methods available through the service should be available, as well as the arguments that can be passed to each method. The interface for WCFStorm is similar.

After the web service has been tested and the data returned has been verified, you can create an ECT using the WCF service rather than a SQL Server view as explained in the previous SharePoint-CRM integration blog post. You can also use the web service as a service reference in another .NET project – with any framework version.

The post SharePoint of Contact 2: Integrating SharePoint 2010 with CRM Online appeared first on DMC, Inc..

Hardware Selection

In order for our robot to provide a lasting platform for autonomous development, DMC wanted a platform that would allow great expandability as well as a very open source structure so as not to limit the robot’s future capabilities. Also, since this project would be starting as a one-day initiative it was decided to keep entry costs low and leverage several commercial technologies to allow a viable product after one day of work.

DMC came up with a simple set of parameters for hardware selection:

Expandability

• DMC will be continually adding features and does not want to be limited by the base platform

Standard communication protocol

• A well-documented command set over serial would save initial setup time and increase compatibility with different hardware and programming languages.

Wireless connectivity

• This will allow large amounts of processing power from a central command server without having to worry about power consumption and allow the robot to freely navigate the office.

Commercial product

• Although this may limit some expandability and configurability, a commercially available platform will save significant hardware setup and debugging time.

Streaming video

• Image processing provides a large range of automation possibilities as well as unique content generation for DMC

Affordability

• The goal was to keep the initial cost of the robot around $500

Based on the above criteria, DMC chose to center the autonomous robot project on three primary components

IRobot Create

The IRobot Create is the development version of the popular Roomba automated vacuum. It includes all the functionality of the Roomba minus the vacuum and adds several additional sensors and a much more expandable I/O configuration. 

DMC chose this for their robot platform for several reasons. The product is well supported by IRobot and has a lot of expandability out-of-the-box. IRobot also supplies a well-documented command set and SDK. Additionally, since it is a well-known product, there is lots of community support including drivers in Microsoft Robotic Studio.

DMC purchased the IRobot Create development kit which includes the robot, a docking and charging station, high-capacity battery and several other components. Additionally, a BAM Bluetooth module was also purchased to provide wireless serial communication with the robot.

Foscam 1918FW Wireless IP Camera

Since it was decided to concentrate processing on a central command server, it was necessary to provide streaming video from the robot. Several companies offer wireless IP cameras that connect to a local Wi-Fi network and can stream video to any computer on the network. A Foscam 1981 was chosen due to its availability, price, and ability to be powered from standard 5V dc.

RoboRealm

RoboRealm is a third-party vision software program that provides easy access to a wide array of vision algorithms and supports plugins to control a large amount of hardware. Additionally, the software provides basic web hosting and a well-documented API for use in other languages. This software was chosen due to its short learning curve, great out-of-box support, and the adaptability to several languages or shifts in designs.

Building and Programming the Robot

Following the included documentation, the IRobot Create was set up to communicate across its Bluetooth network with the host PC. The Foscam camera was also set up on DMC’s local network using a LAN connection and then configured to connect to the wireless network on power-up. 

Once configured, the camera was attached to the top of the IRobot Create and connected to the robot’s switched battery power supply using a 5V regulator. This was necessary since the robot’s 5V outputs were not capable of providing the current necessary to power the camera.

DMC leveraged the out-of-box functions in RoboRealm to create a working robot in a short timescale. The programming module involves a scan cycle based on the incoming frame rate from the chosen camera. Different modules can be dragged and dropped into this scan cycle, and they include image manipulation algorithms, command modules, general logic functions, and scripts in several languages. 

Once placed in the processing pipeline, the modules can be configured based on their specific function. The software comes packaged with plugins for both the IRobot Create and Foscam camera that handle the low-level communication with each device and allow commands and statuses to be sent/read using user-created variables. Below is an image of the three primary modules used in the autonomous robot as well as the processing pipeline.

We can now go into detail on each of these areas, starting with the image stream.

The image stream shows the current video stream from the chosen camera, in our case, the Foscam’s wireless stream. The image stream also shows the results of all the image processing up to the selected step in the processing pipeline. Additionally, if the web server is set up, this is the same image that is streamed to the site.

Next, we will look at the processing pipeline.

This is where the overall robot functionality and image processing is defined. Functionality is added by placing modules into this pipeline and programming their functionality. 

At the very top, you see the programming block for our wireless camera. This block allows the address of the video stream to be defined along with login information for the camera’s controls. It also allows a variable to be set controlling the pan and tilt of the camera. 

The majority of the processing pipeline after the camera is devoted to three different operating modes. Within each operating mode, there is a set of image processing modules that return a set of variables derived from the incoming image stream as well as a VBScript module that computes robot commands based on these image variables. Lastly, there is the IRobot create a module that defines several variables used in the script modules to control aspects of the robot.

We will now look at the script module in more detail as this controls most of the robot’s operation logic.

The scripting module allows you to write scripts in several languages (visual basic in our case) to provide logic for your processing stream based on variables defined in other modules. In our program, these scripts are used to take data from the image processing and robot sensors and use it to send movement commands to the robot. 

For example, most of the image processing produces x and y coordinates of a targeted item on the image (line, farthest clear path, etc.). These coordinates are then used to set the variables defining the robot’s left and right motor speeds to different values telling the robot to move forward, backward or turn. 

Other variables representing the robot’s sensors can also be queried to help control the robot (e.g. when the front bump sensor detects an object and the robot moves backward).

Lastly, we will look at both the camera and robot modules.

These modules don’t provide any specific programmability, but rather allow variables to be assigned to different functions of the robot and camera. By including them in the processing pipeline, they also handle all low-level communication with each device to constantly update variables and send commands.

Web Deployment

Finally, to allow control of the robot by any computer in the office, the robot’s controls were deployed to an internal website using RoboRealm’s web hosting feature. By activating the web server feature, RoboRealm publishes a website on the host computer using files customizable by the programmer and a JavaScript interface to send and receive variables from the web page. 

The web page was set up to allow manual control of both the robot and camera as well as activation of the two autonomous modes and the IRobot’s default docking program. It also updates the user with the state of various sensors on the robot as well as battery levels. DMC leveraged an example page from RoboRealm’s tutorials to facilitate the creation of a functional and appealing web page.

Be sure to check back for continued development of the DMC autonomous robot!

The post Office Autonomous Robot appeared first on DMC, Inc..