Archive for the ‘Tools’ Category

A nice coincidence a few weeks was the news of Raygun going in to public beta crossing my radar.

At the time we were fine tuning some things in an application that was in a private beta, we had put a little effort in to ensure that we would get reliable results about errors that happened to the users, but at that point we were just storing the details in a database table.

Background

We were capturing 3 levels of errors in the application.
– Client-side (JavaScript)
– Web Tier (ASP.NET MVC / WebApi)
– Back-end (Topshelf hosted services)

Any client side error would be captured, and sent to the Web Tier, Web Tier forwards that and it’s own errors on to the back end where they would be persisted with low overhead. In a previous post I have covered this approach.

But to get from entries stored in a database to something actually useful to correctly monitory and to start a resolution process is quite a bit of work.

From our own application structure; we can easily query that table, and just as easily send emails to the dev team when they occur. But this is still short of a robust solution, so a quick glance at the Raygun features and there was very good reason to give it a go.

What it took for us to set up Raygun

A quick look at the provided setup instructions and their github sample, it looked very easy.

With our particular application structure the global Application_Error method and the sample usage of Server.GetLastError() didn’t fit well. The clearest example is the arrival of data from client side, which isn’t a .NET exception, so simply issuing the RaygunClient().Send(exception); call doesn’t work. In this scenario we basically recreate an exception that represents the issue in the web tier, then have that sent to Raygun.

For errors that originate in our controllers (regular and WebApi) which extend a common base class, we make use of the HandleError attribute so we can execute a method to do some extra work, the code looks like:

[HandleError]
public abstract class BaseController
{
    protected override void OnException(ExceptionContext filterContext)
    {
        //our other logic, some to deal with 500s, some to show 404s

        //make the call here to raygun if it was anything but a 404 that brought us here.
        new RaygunClient().SendInBackground(filterContext.Exception);
    }
}

In the scenarios where we actually do have the exception, then it’s great and it “just works”, and we send it off asynchronously, in the catch block by calling a wrapping function like this:

public static void LogWithRaygun(Exception ex)
{
    new RaygunClient().SendInBackground(ex);
}

Conclusion

So Raygun really helped us avoid using a weakly hand-rolled half-way solution for tracking errors, now with nice email notifications that look like this, and link into the Raygun detailed information view.

It’s lacking a few nice to have features, but that’s more than acceptable for version 1 of the application, and from what we’ve been told our suggestions are already on track for a future release. One particular one that would benefit lots of people would be to allow an association of errors to be mapped by the user. An example is, 2 seemingly different errors get logged but in actual fact are the same cause, this way the reporting and similarity tracking can continue to group the 2 variations under the one umbrella.

raygun email example

Along with the dashboard summary.

Part of the Raygun  Dashboard

It’s one less thing we need to worry about. Just an FYI we didn’t stop saving records into our own database table, we’re just unlikely to have to go looking in there very much, if ever.

Mailzor is a basic utility library to help generate and send emails using the Razor view engine to populate email templates, designed to be quickly pluggable into your .NET app.

In our applications we send out HTML formatted emails, and seed them with a variety of data. I thought it would be easy to write them as razor files (cshtml) and then use the razor engine to generate them and send.

It’s up on NuGet and with the release of v1.0.0.11, it’s more stable.

For the most up to date info follow along with the usage sections of the readme.md file on the github repository.

How it works

I thought I would share some background about the development of it, and hiccups along the way. The original set of code came from Kazi Manzur Rashid, which solved the problem of making use of System.Web.RazorTemplateEngine, which I extended (with permission) to be usable as an injectable dependency and via NuGet.

The core elements are, the creation and management of the SMTP client, the building up of the MailMessage. Then all the compilation related work to get the RazorTemplateEngine up and running.

The RazorTemplateEngine logic boils down to taking the razor file stored on disk and using CSharpCodeProvider.CompileAssemblyFromDom. So if you’re curious about this code in particular dig into EmailTemplateEngine.cs in the project files.

Prior to version .10 where I went down the path of using ilmerge to solve conflicts of version mismatch with System.Web.Razor.

It seems easy seeing how I took an existing chunk of operational code and extended, and it only seems easy when it is working, but when it doesn’t work and you’ve got strange compilation errors, debugging this mechanism is not the greatest. I found myself hunting for temporary files and trying to have other compiler flags to output more information.

In the early versions it was heavily the case of “works on my machine”, but now its fine and seems to be feature complete…

I’m working towards a more complex SignalR based post, but in the mean time part of the work on that involves setting up a few ASP.NET web apps.

If you’re after a more comprehensive guide check out this post on learn.iis.net by Thomas Deml. I’ve summarised the steps required to get some basic .NET 4 web applications deployed.

Objective
To create N identical websites in local IIS, each with an incrementing name Id. Each linked to the same application directory. The exact reason as to why will come in a future post, for now consider it an exercise in manipulating IIS via PowerShell.

s1.site.local
s2.site.local

Step 1 – Ensure you’re running the scripts in x86 mode.

Which seems quite common a problem, with a stackoverflow question. I haven’t worked a way around this yet, but this is the error when not running as x86:

New-Item : Cannot retrieve the dynamic parameters for the cmdlet. Retrieving the COM class factory for component with CLSID {688EEEE5-6A7E-422F-B2E1-6AF00DC944A6} failed due to the following error: 80040154.
At line:1 char:9
+ New-Item <<<< AppPools\test-app-pool
+ CategoryInfo : InvalidArgument: (:) [New-Item], ParameterBindingException
+ FullyQualifiedErrorId : GetDynamicParametersException,Microsoft.PowerShell.Commands.NewItemCommand

Step 2 – Import-Module WebAdministration
This loads the IIS namespace which allows you to just navigate in the same way you would the filesystem

> CD IIS:\

Step 3 – Create & Configure App Pool

    New-Item AppPools\test.pool

    Set-ItemProperty IIS:\AppPools\test.pool -name "enable32BitAppOnWin64" -Value "true"

    Set-ItemProperty IIS:\AppPools\test.pool -name "managedRuntimeVersion" -Value "v4.0"

NOTE: here I didn’t have any luck storing the app pool path in a variable then using Set-ItemProperty, hence the repeating.

Step 4 – Variables

    $sitesToCreate = 10
    $path = "C:\dev\project-x\App.Web"
    $appPool = "test.pool"

Step 5 – Create Site & Link AppPool Loop

For N(10) to zero:

  • Create a new site
  • Set binding info and physical path
  • Set the app pool
    while ($sitesToCreate -gt 0)
    { 
        $siteName = "s" + $sitesToCreate + ".site.local"
        $siteWithIISPrefix = "IIS:\Sites\" + $siteName
        Write-Host "Creating: " $siteName
        
        $site = New-Item $siteWithIISPrefix -bindings @{protocol="http";bindingInformation="*:80:" + $siteName } -physicalPath $path
        
        Set-ItemProperty IIS:\Sites\$siteName -name applicationPool -value $appPool
        $sitesToCreate--
    }

Note: ‘appPool’ is a text variable, not the ‘Get-Item’. Set-ItemProperty operates on a path not a variable representing the item.

We’re done

One last note, to get these to resolve correctly on your local developer machine you’ll need to modify your hosts file.

IIS multisite

The complete code up as a Github Gist.

Wow that’s one hell of a title, I couldn’t make it any shorter, but that’s everything we’re dealing with in this post.

Some Background

We’ve got an ASP.NET MVC 4 web application and we’re using Cassette to bundle and minify the JavaScript and CSS files, the reason is that the new Beta 2 MVC 4 bundlers didn’t work right away, and we already had Cassette all configured.

Cassette further solidified its place as a tool of choice for us when it introduced Jasmine bundling. We can just point a Cassette bundler at a location in the project that contains the spec files (jasmine tests) and be done, not have to worry about wiring up any spec helper files with paths, etc. The current version (1.2.0) which is great, but does not handle CoffeeScript compilation in an efficient way, yet…

Only have to remember to do the standard approach of using

/// <reference path="your-js-file.js"> 

Wanting to use CoffeeScript

Now we’re also jumping on the CoffeeScript bandwagon *because it’s just JavaScript*(^TM), and we want our code to be neat, elegant and tested, and CoffeeScript looks like it will help out. The last thing that was holding me back on going all in on CoffeeScript was getting it smoothly into the Visual Studio or our build process (if also necessary).

Turns out that is very easy if you just use Web Workbench from Mindscape. It’s a free extension for Visual Studio. Go here for their getting started guide.

How we’re using Web Workbench

Our primary desire is just to have the CoffeeScript compiled for use without fuss. Including speed. So Web Workbench helps in this department, also the creation of the .js files nested under the CoffeeScript files. We are just writing our client side code in CoffeeScript and want the process as simple as possible.

The highlighting will be helpful if still in Visual Studio, and will be helpful to us when it’s working in the Visual Studio 11. We’re spending a reasonable amount of time in VS11 already.

As I mentioned earlier Cassette takes care of “packaging up” all our JavaScript files already, so to combine our newly automatically compiled CoffeeScript logic it was as simple as setting up a Cassette bundle pointing to the location of the Jasmine spec files.

Download the Web Workbench here.

The Cassette side of the story
The most basic configuration to get the Jasmine tests bundling and set up to run with the least effort is to just tell Cassette where they are wit this line:

   bundles.Add("JavaScript/Specs");

That snippet lives inside the CassetteConfiguration.cs file installed by its NuGet package.

public class CassetteConfiguration : ICassetteConfiguration
{
   public void Configure(BundleCollection bundles, CassetteSettings settings)
   {
      //all the other cassette configuration code
   }
   bundles.Add("JavaScript/Specs");
}

Putting it all together

So here’s what the CoffeeScript code looks like inside Visual Studio (v11).

The actual function we’ll be testing:

The test:

Here’s how Web Workbench presents the CoffeeScript files with their compiled JS:

The test runner, and the url path, this is what Cassette helps make simple:

Summary

That’s it. So we’ve written some logic in CoffeeScript, and then unit test it with a Jasmine spec also written in CoffeeScript, Web Workbench handles the compilation as we type, and Cassette puts it all together to display and run in the browser via the bundle url.

If you want to see the Jasmine side of things in action (the repository doesn’t have the CoffeeScript changes finalised yet), check out the sample code up on this GitHub project.

We’ve gone to some lengths at work to automate, and have a continuous delivery style pipeline for all things build and deployment related. It’s well on its way, but not ‘perfect’ yet. Maybe perfection isn’t attainable, or maybe when you have a red button on the desk that when pushed does *everything*. None-the-less aiming for perfection will continue to drive us to improve.

So here I wanted to discuss what steps I took to get a nice JSLint Visual Studio plugin to form part of our build process. It was a bit of fussing about getting it to work, and it’s an example of still being imperfect, but for now it serves the build pipeline well enough.

If you don’t use JSLint for your JavaScript code, you probably should. It’s a static analysis tool to analyse and report on broken coding rules for JavaScript. Try it out on the creator (Douglas Crockford’s) site JSLint.com.

There is an easily accessible Visual Studio plugin called exactly what you would expect “JSLint for Visual Studio 2010” and here is the direct link for it on the VS Gallery for your installation pleasure.

If you do anything and stop reading here you’ve done well, install it and happy JavaScripting.

JSLint for Visual Studio 2010 Extension

But what about continuous integration?

For us it wasn’t enough, we wanted our build process which runs as psake scripts to fail if JSLint rules were broken.

Just for a bit more completeness on the psake digression and how the command line tool will execute under MSBuild, here’s a snippet of the ‘targets’ code that is referenced by the .csproj files that contain JavaScript. The input parameter on the executable being the directory containing the script files relative to .csproj file.

<Target Name="AfterBuild"> 
    <Message Text="Running jslint..." Importance="Normal" /> 
    <Exec Command="&quot;..\jslint\JSLint.CLI.exe&quot; .\Scripts\ " /> 
</Target>

So the search for command lines tools began. I found some existing command line tools, some which had more complex dependencies, others which seemed to be more ‘primitive’, i.e. did not report on errors the IDE based JSLint plugin was reporting.

There were 2 main objectives driving the choice of the tool

  1. Similarity and accuracy to JSLint.com and the Visual Studio extension
  2. Ease of setup

There was some discussion happening on StackOverflow, here and here but nothing I tried digging deeper into seemed suitable.

I then had an idea…

Take the core logic of the Visual Studio extension and wrap it in a very simple console application to execute as part of the build process.

With the approach I took, it seems even option 2 was difficult to achieve (consumed some time), but at least it had the least external dependencies to the other options out there.

The very first step was to obtain and install the VS2010 SDK, as this was an project which referenced many interop assemblies for interacting with the IDE. It needs to be the Service Pack 1 SDK in fact, and here’s a direct link. Once I was able to compile the extension it was then a matter of understanding how it operates and how to access a method to perform the ‘linting‘.

There were 2 major hacks to get access to some inner workings of the extension to operate:

  1. Making some ‘protected/internal’ methods and properties public
  2. Modifying where the JSLint logic obtained the settings file from (JSLintOptions.xml).

Locating JSLintOptions.xml proved somewhat difficult at first, as it was tucked away hiding in the Roaming section of my user folder on Windows (\Users\*\AppData\Roaming\). These hacks could greatly benefit from some re-factoring effort if I have ever have the time, or someone else is so inclined. But after an initial attempt to refactor out the most core of logic things fell apart in the land of SDK dependencies, so I rolled back and opted for a less elegant approach which was the 2 hacks listed above.

The logic for the console application is then trivially simple:

  1. Read .js files.
  2. Using the JSLinter class method Lint() supply the .js file content/
  3. Write errors to console
  4. Return error code, 0 (Success), 1 (JavaScript warnings), etc.

Show me the code!

If you want to see the hacks I had to make to get this to work head on over to this git repository on BitBucket. I offer no warranty it may be fragile so manipulate with caution.

If you just want the command line executable, then it’s built and stored in the repository in the /output folder, if I make any updates to the executable, I’ll also update that executable.

Areas for improvement in the console app:

  • Taking path locations as input parameters/external file of locations
  • Taking alternate settings files for rules to ignore/include
  • General tidyup

Outstanding issue

The final hiccup is not directly related to the use of the command line tool itself, but building the command line tool from source as part of a more comprehensive-all-inclusive build process. Adding commands in the .csproj file post-build settings was not sufficient. The build and copy of the executable needs run as a directive in the targets ‘afterbuild’ section of the .csproj file.

CSProject Settings Dialog - Post Build Events

Not suitable to place actions in 'post build' section

This led to a conflict between MSBuild and the VS2010 SDK, which was very frustrating and currently isn’t solved yet. The question is up on StackOverflow.

Over the last few months at least in the streams of information I typically consume, direct issues: Security Now topic of Password Haystacks, xkcs’s comic, Coding Horror, and indirect: Scott Hanselman one and two. Have all commented on the issue around passwords and strength and the need for better passwords.

In this post I am putting forward a novel approach: which as an homage to GRC’s Perfect Paper Passwords and accordingly have titled my approach:

When high entropy 16, 32, 64 or even 128 character passwords are just not secure enough!

Let’s jump right in with a sample, here I’ve mocked up the very familiar facebook interface with a nice large textbox to put in your Perfect Password Paragraphs™.

Perfect Password Paragraphs facebook log in modified

Disclaimer: if you’ve gotten this far and haven’t begun to appreciate the humour I’m so sorry, please don’t send me hate mail.

Features:

  • A big text area where with probable difficulty you have to type 100+ words to authenticate.
  • Typographical errors are ok as long as they are consistent for you.
  • A flow of sentences following a theme/style just needs to sound like the individual attempting to gain access.
  • “Sound Like” is a trademark (patent indefinitely pending) of Josevski Research Corp, is the flux capacitor grade specialty of this authentication system.

Comparison metrics:

  • Writing style
  • Choice of punctuation, frequency of commas, periods, ect.
  • Grammar choice.
  • spelling (American vs British English).
  • Consistency of spelling errors.
  • Choice of tense (present, past, and future)

Future Features based on demand:

  • International support.
  • 1337 sp34k.
  • Baby talk.
  • Obscure localised slang.
  • Pig Latin.
  • iOS, Windows Phone 7 and Android Support.

Alpha product coming online in 6-8 weeks ;)

On a large repository, I was attempting to rename the SVN tracking folders that are nested at every directory level, I needed to do this because of a difference in the leading character ‘.’ (period) vs ‘_’ (underscore). I know this could have easily been resolved with a new fetch but I wanted to avoid a lengthy download over a VPN connection.

I thought I would just quickly list some PowerShell commands I was playing with to clean up the repository as a blog post.

The closest I got to a solution but with a lot of errors/warnings during the process was:

Get-ChildItem * -Recurse -force | Where-Object { $_.Mode -eq "d--h-" } | Rename-Item -force -newname '_svn'

It seems the -force parameter was required. I’m not sure why it errors but it still works. Further investigation would be around how many times the commands run per directory, possibly too many times. Another avenue for investigation is the -silent parameter but that’s likely only going to obscure any issues.

Just for reference here’s what else I tried, these did not succeed.

Get-Childitem -path . -include .svn -recurse | Rename-Item -newname {$_.name -replace '.svn','_svn'}
Get-Childitem -path . -include .svn -recurse | foreach { Rename-Item .svn _svn }
Get-Childitem -path . -recurse -include '.svn' | foreach { Rename-Item .svn _svn }
Get-Childitem -path . -recurse | rename-item -newname { $_.name -replace '.svn','_svn' }
Get-Childitem -path . -recurse -include .svn | move-item -destination _svn

If you’re a PowerShell expert feel free to correct my possibly misguided attempt at a recursive rename.

Update 29th Dec 2011
I stumbled upon someone much more clever undertaking a similar rename process. In this case jQuery text, but the logic serves the same purpose it goes and renames content of the files that logic can be replaced with the move-item command.

$find = 'jquery-1\.4\.4'
$replace = 'jquery-1\.5\.1'
$match = '*.cshtml' , '*.vbhtml'
$preview = $true

foreach ($sc in dir -recurse -include $match | where { test-path $_.fullname -pathtype leaf} ) {
    select-string -path $sc -pattern $find
    if (!$preview) {
       (get-content $sc) | foreach-object { $_ -replace $find, $replace } | set-content $sc
    }
}

Just as a quick update post on my on going series of posts on using PLINQ on Stack Overflow data-dump.

In my initial post where the core of what I was doing was outlined, at the time the popular (and quickly found) option was to use a series of stored procedures made available by Brent Ozar to import the XML data into a SQL database.

XML into DB

Brent recently replied back on the original post tipping me off to an easier more convenient way to get the data into SQL.

… There’s an even faster way to import the XML files now using Sam’s SoSlow.exe tool. You give it a connection string (including the database name) and it’ll create the tables and import the data. Just FYI – it doesn’t warn you, but it does delete and recreate the import tables every time. It’s dramatically faster too.

I’m all for an “easier” and “better” approach, so I gave it a try.

The first step was to get a copy from Sam Saffron‘s GitHub respoistory

http://github.com/SamSaffron/So-Slow/downloads

It is a small C# WinForms application with 3 buttons, so the use of it very simple and suits well with the also simple layout of my PLINQ demo application.

SoSlow Interface

In under 15 minutes all the data was imported (results will vary depending on your machine configuration). This will help out keeping the data more up to date when the next public release of the data is made available.

StackOverflow DataBase Successfully Created

I just got home from the attending the October Silverlight Designer and Designer Network (SDDN) meeting. It was a trillogy presentation from the Readify guys, Jordan Knight, David Burela and Philip Beadle.

Who respectively presented on:

  • Connecting Silverlight to RIA,
  • Binding data to Bing Maps and
  • A testable Silverlight architecture and development approach walk-through

There was also an announcement for the 2010 Melbourne Silverlight CodeCamp, with registrations now open at the SilverlightCodeCamp.com.au. The schedule is still open for volunteers to offer up presentations. It will be run at the new NAB training facilities. The registration site was mentioned with the footnote of “it’s not tested yet” which at the time of this post seems to be the case, my registration isn’t being processed correctly or at least there’s no feedback of success.

First up Jordan ran through the basics of using .NET RIA Services to produce a simple data-bound Silverlight website log-in component. It included demonstrating how to use attributes to decorate the RIA based entities to assist with validation that can be shared server side and client side. The benefit being a visually well designed Silverlight application can benefit from robust validation logic, and make use of it through field level bindings to easily display validation feedback to the user.

Next up was a quick presentation from David showing us data binding capabilities of data collections to Bing Maps, to produce overlays and interactive features on a map view. He should be posting the material up soon on his blog, including some code snippets that will make achieving what he demonstrated quick and easy.

The final presentation from Phil, which was nicely presented using Prezi.com was all about putting together a collection of tools and concepts to develop a robust and testable Silverlight based application. It focused around using the Model-View-ViewModel (MVVM) architecture/design pattern. It included examples of regular unit tests, automated UI tests, and integration tests, making use of an Inversion of Control (IoC) framework called Unity a Microsoft Application Block up on CodePlex and the Visual Studio 2008 Test Edition. There’s a bit of a discussion going on at StackOverflow about Unity.

Some key things that Phil pointed out to stay up to date with Silverlight and of course to assist with building more testable Silverlight applications include:

All up, a good round of presentations.

Full disclosure: This is way off topic for my blog and I just wanted an excuse to embed a Simpsons episode in a powerpoint presentation.

Inspired by this post on the PowerPoint MSDN Blog and its associated YouTube video, demonstrating the ability of PowerPoint 2010 to use DirectX and hardware support for its rendering engine. Great effects can be achieved smoothly. Like a lot of us (most unwilling to admit), I’m impressed by reflection of a video as it plays.

Slide Show Preview

Slide Show Preview

Setting up the 3D Rotation Effects were also easy:

Format Video

Format Video

Same goes for the Reflection effect:

Format Video Reflection

Format Video Reflection

Enjoy:

Reflection Overflow

Reflection Overflow

Now saving this pptx file showed me that in-fact the video file is embedded, so it’ll make moving the pptx file easy (so long as it doesn’t require to be emailed).

File Size

File Size

Once a video is added you can make use of it on several slides, and use the ‘Trim Video‘ feature to select a section of it for each slide.

Trim Video

Trim Video

2 Slides each with a segment of the video:

Reusing Video

Resuing Video

The presentation file along with the video that was embedded, there is only 1 copy of the video in the pptx file.

File Size of Multiple Video Copies

File Size of Multiple Video Copies