byte-order marks are evil, dangit!

2009-01-29 @ 22:24#

ok, got bit in the butt by this one this week. and it hurt bigtime. cuz it showed i was a dope (again).

here's the run up to my latest humbling:

i've been doing quite a bit of HTTP programing using C#. to make it easier to deal w/ i wrote an HTTP client library that encapsulates most of the hard stuff. now i can do simple stuff like this:

client.RequestHeaders.Add("authorization", "Basic " + h.Base64Encode(string.Format("{0}:{1}", user, password)));
client.RequestHeaders.Add("cache-control", "no-cache");
rtn = client.Execute(url, "get", "text/xml");

and all the other cool stuff like POST, PUT, DELETE, HEAD, OPTIONS, et al. very nice i can even pull back a bit stream to handle binary responses like images, PDFs, etc.

client.UseBinaryStream = true;
ms = new System.IO.MemoryStream();
client.Execute(url, "get", "image/png", string.Empty, ref ms);

and that last little ditty got me in trouble.

see, i could handle simple text responses via the binary stream, too. no reason to do anything diff, right? so i got into the habit of doing this:

client.UseBinaryStream = true;
client.Execute(sds_url, "get", accept_type, string.Empty, ref ms);
System.Text.Encoding enc = System.Text.Encoding.UTF8;
rtn = enc.GetString(ms.ToArray());

yeah, did you see that? keep the last three lines in mind.

here comes the part were i get pwned

this week, the cool folks on the SDS team did us all a favor by updating the service to fix a few small items. one of the things they added (for free) was the ability more easily support multiple encoding types for returned Entity XML. you know, support for UTF-16, UTF-32, etc. very cool. very unexpected. very "break-mike's poorly-written apps."

see, i wrote all my code *assuming* no silly stuff like added byte-order marks(BOM) in the response. !dumb! now all my code that tried to load the returned stream, convert it to a UTF-8 string and then push it directly into an XMLDocument object blows up. cuz i assumed -waaay- too much.

really dumb part? the XMLDocument object can sort this out on it's own, if i would just get the hell out of the way. like this:

client.UseBinaryStream = true;
ms = new System.IO.MemoryStream();
client.Execute(url, "get", Constants.SsdsType, string.Empty, ref ms);

yep. i am humbled. again. thanks to the SDS guys for keeping me honest. really.