Yesterday, I posted the Screen Scraping Utility Toolkit to MSDN.

Here’s why you’ll want to take a look at it:

1. You deal with unstructured data.
2. You have legacy data stored in HTML and want to normalize it or put it into objects
3. You want to create an app for a web site and you (or they) don’t have an API

In a post from late last year, I posted the Screen Scraper’s Manifesto, which was meant to be be a teaser for this very utility kit.

This is how the utility kit works: it provides a base class that wraps up the WebClient class and an easier means to parse values out of text.

There’s also a way to offset the risks inherent to screen scraping: changes in the target site.

In this first post, let’s talk about text processing utilities.

All this this magic happens in the ExtensionMethods class in the Utils directory with the FindElement and FindElements method.

As you can see the code below isn’t too complex, but what it does is save you a lot of work.

        public static string FindElement(this string sourceString, string startDelimeter, List<string> endDelimeters)

        {

            int endPoint;

            return FindElement(sourceString, startDelimeter, endDelimeters, out endPoint);

        }

        public static string FindElement(this string sourceString, string startDelimeter, List<string> endDelimeters, out int endPoint)

        {

            var startPoint = sourceString.IndexOf(startDelimeter);

            if (startPoint == -1)

            {

                endPoint = -1;

                return null;

            }

            var adjustedstartPoint = startPoint + startDelimeter.Length;

            endPoint = FindEndElement(sourceString, endDelimeters, adjustedstartPoint);

            var length = (endPoint - adjustedstartPoint);

            var subString = sourceString.Substring(adjustedstartPoint, length);

            return subString;

        }

If you’re a seasoned developer, you have no doubt have already wrote code to pull out values from strings. Think of all the code you’ve had to do for that.  Now look at this bit of code, which uses extension methods, to make the code compact.

var eventsToday = this.RawText.FindElement("Events</span></h2>", new List<string>() { "<h2>" });

This takes the HTML from a typical “events occurred on this day” pages and pulls out only the events portion of the HTML. To split out the individual events, we’ll use the other method: FindElements

        public static List<string> FindElements(this string sourceString, string startDelimeter, List<string> endDelimeters)

        {

            List<String> returnList = new List<string>();

            List<int> length = new List<int>();

            int totalLength = sourceString.Length;

            int currentPoint = 0;

            int endPoint;

            string workingString = sourceString;

            while (workingString.Length > 1)

            {

                workingString = workingString.Substring(currentPoint);

                int workingStringLength = workingString.Length;

                if (workingStringLength == 1)

                {

                    break;

                }

                length.Add(workingStringLength);

                string element = workingString.FindElement(startDelimeter, endDelimeters, out endPoint);

                if (element == null)

                {

                    break;

                }

                currentPoint = endPoint;

                returnList.Add(element);

            }

            return returnList;

        }

This means you can extract each of the events with one line of code:

var events = eventsToday.FindElements("<li>", new List<string>() { "</li>" });

The line of code above breaks apart the items on the events list into a List of type string. From there, you can slice and dice the data more.

But the ugly parts are done for you and you can focus more on the big picture.

Now, if only there were an easier way to grab text from the web.

Stay tuned for my next post.

Here’s another great episode of .NET Rocks.  In show 737, Carl and Richard talk to Jeremy Likness about Silverlight 5.

It contains some great content on the new features of Silverlight 5 and the question that’s on everyone’s mind.

Carl and Richard talk to Jeremy Likness about Silverlight 5. Is Silverlight dead? Not by a long shot. Jeremy talks about the great work going on the Silverlight today and how it still is the most efficient way to build applications that run on both Windows and OSX. The conversation drills into the new features of Silverlight 5 and dispels a lot of the myths around the future of Silverlight. Jeremy also talks about Jounce, his MVVM+MEF framework on Codeplex and the relationship between Silverlight and WinRT.

Jeremy Likness is the author of “Designing Silverlight Business Applications” and was named Silverlight MVP of the Year in 2010. Now Senior Consultant and Technical Project Manager for Wintellect, LLC, he has spent the past decade building highly scalable web-based commercial solutions using the Microsoft technology stack. He has fifteen years of experience developing enterprise applications in vertical markets including insurance, health/wellness, supply chain management, and mobility. He is the creator of the popular MVVM framework Jounce and an open source Silverlight Isolated Storage Database System called Sterling. Likness speaks and blogs frequently on Silverlight, MEF, Prism, Team Foundation Server, and related Microsoft technologies.

When I arrived at my hotel for the SC-GMIS Developer Summit, I saw a travel brochure for a nicely named establishment: Frankie’s Fun Park.

With a name like Frankie’s, it’s has to be good.

So, on my way back to the airport, I stopped in to check the place out and pick up some “Frankie’s swag.” After all, it had my name all over it.  

Last week, I had the pleasure of speaking at the SC-GMIS Developer Summit in Columbia, SC.

As promised here are my slides:

Several attendees also asked I share a list of resources for those just getting started in .NET and I thought I’d repost the list here so all may benefit.

• Getting started with C#
• http://msdn.microsoft.com/en-us/library/a72418yk.aspx
• http://www.csharp-station.com/Tutorials/Lesson01.aspx
• http://www.homeandlearn.co.uk/csharp/csharp_s1p1.html (screenshots are from VS 2008 era, but content still good
• http://ondotnet.com/pub/a/dotnet/excerpt/learningcsharp_2/index1.html
• ASP.NET MVC 3 Intros
• http://www.asp.net/mvc/tutorials/getting-started-with-aspnet-mvc3/getting-started-with-mvc3-part1-cs
• http://www.asp.net/mvc/videos
• Getting started with Silverlight
• http://www.silverlight.net/getting-started
• Columbia, SC .NET User Group
• http://columbiadevelopers.org/Home.aspx
• Podcasts
• http://www.dotnetrocks.com/
• http://www.communitymegaphonepodcast.com/
• http://hanselminutes.com/
• Subscription training
• http://tekpub.com/
• http://www.pluralsight-training.net/microsoft/
• Blogs:
•                 Rachel Appel http://rachelappel.com/
•                 Andrew Duthie  http://devhammer.net/
•                 Roberto Hernandez http://blog.overridethis.com/
•                 Scott Hanselman http://www.hanselman.com/blog/
•                 Pete Brown http://10rem.net/
•                

This list (especially the blog list) is by no means complete.  If you have some suggestions, please add them to the comments.

Thanks!

SOPA and PIPA may have been tabled for now, threats to our online freedoms are still out there.

Here’s an infographic on the two (currently) dead bills:

[found via AllTop]

It’s a set up for a Volkswagen commercial that will premiere during the Super Bowl and mighty darn cute if you like dogs or Star Wars.

Here’s a great example of how the Silicon Valley style of entrepreneurial spirit, innovation, and opportunities has spread throughout the world.

social management blog

[found via AllTop]

Visit http://stopthewall.us for more information.

WikiPedia is shutting down today to protest the proposed SOPA/PIPA laws.

While every reasonable person agrees that piracy must be stopped, this legislation is not the answer.

The text of the law has some scary provisions.

This is my opinion and not necessarily that of my employer, its business partners, or even my dogs.

Mary Sue showcases the work of “paperbeatsscissors” where the droids of the Star Wars universe share their feelings.

As an added bonus: here’s Darth Vader if he were into hip-hop:

I just posted an update to my Texts From Last Night application that clears up an ugly text encoding issue.

Version 1.3 should be available in a few days.

Thanks!

My team at Microsoft is teaming up with ComponentOne to deliver a series of webcasts to demo how easy it is to build Public Sector mobile solutions on Windows Phone.

1. Session I: Fast Track Your First App
2. Session II: Give Your Apps Some POP

Be sure to register today.

[found via Joel Reyes]

This Thursday, Jan 19 at 7PM, the CapArea Silverlight SIG presents Franklin Balaquer on XNA on Windows Phone.

The XNA Framework and Windows Phone 7 - Franklin Balaguer

Thursday, January 19, 2012 at 7:00 PM

Abstract:

XNA Game Studio and the XNA Framework are designed for cross-platform gaming scenarios with support for Windows Phone 7, Xbox 360, and Windows-based PCs. This allows you to target more platforms from the same code base. XNA Game Studio includes the XNA Framework, a set of managed libraries designed for game development based on the Microsoft .NET Framework.
In this workshop, I will go over some free resources available to help you create your own Windows Phone 7 game and then sell it on the WindowsPhone MarketPlace. I will also touch on the basics of XNA (drawing a sprite, handling of input touch, playing sounds, and game state management). Lastly, I will discuss the certification process for getting your game approved and placed on the MarketPlace.

Bio:

Franklin Balaguer is U.S. Air Force veteran. He spent 8 years in the Air Force managing vehicle fleets at various bases in the United States and while deployed overseas. Frank got his start with computers writing programs to automate tasks and processes everywhere he encountered them. While in the Air Force, he also acquired several certifications including the Microsoft Certified System Engineer, which help him land a job at Accenture. Frank has been with Accenture for the past 11 years helping to write and deliver software to thier clients.

Now that IHeartRadio has an app for the XBOX, I can listen to radio stations from around the US, 

 While browsing New York radio stations, I noticed an encoding error.

Zoomed in for detail.

Just goes to show you that everyone has encoding issues.

Carl and Richard have John Papa over to discuss XAML, Silverlight, and the landscap of Microsoft technologies. [download MP3]

Carl and Richard talk to John Papa about his views on the client development landscape. John has recently left Microsoft where he was a Silverlight and Windows 8 evangelist. The conversation starts off talking about the state of Silverlight now that Silverlight 5 has shipped and the ascent of HTML 5. John talks about his favorite HTML 5 libraries, including KnockoutJS and JS Render. Finally, the show wraps up with how Windows Phone 7 and Kinect will change the world.

John Papa is a former Evangelist for Microsoft on the Silverlight and Windows 8 teams where he hosted the popular Silverlight TV show. He has presented globally at keynotes and sessions for the Build, MIX, PDC, TechEd, VSLive and DevConnections events. John is also a columnist for VS Magazine (Papa’s Perspective) and author of training videos with Pluralsight. Follow John on Twitter at @john_papa

Microsoft’s got a large booth at CES, and by large, I mean gigantic.

The booth occupies nearly an acre of convention floor space and it’s filled with some of the hottest tech around. Full coverage on Microsoft’s Facebook page.

Another great animation by RSA to a talk by Sir Ken Robinson, world-renowned education and creativity expert.

Microsoft’s 15 year history at CES, remixed and auto-tuned into a 2:27 minute video.