In order to make Cross Domain calls in Silverlight 2 applications, the server you wish to download data from must have a Policy File in place that allows access to your site.
See Tim Heuer's blog post for more on this and a code snippet utility.
If there are no Cross Domain Policy files or the policy files do not allow HTTP requests from your domain, then you'll need to use a proxy like the one I wrote.