Well firstly I have to say its not that easy , secondarily it really shows the true benefit of [TDD] – more on that later. So on to the discoveries!

Being a TDD advocate I started off with an interface and then the tests before the code. So far so good – then on to the code. As a Spongebob episode would say “Several hours later” we were done. So crank up the tests and we are looking golden.

Unfortunately the code coverage wasn’t half as good as I had hoped for. So next couple of hours was spent updating the tests for coverage, and a number of edge cases surrounding the throwing of exceptions for invalid parameters. Interestingly a couple of areas that showed up outside of the TDD approach were certain areas in “for loops” where not executed because my test case was really unknowingly best case. The part that was revealing to me was that the TDD is really a “business” driven approach, code coverage is believed to be a programmers indicator of  ‘quality’. So while TDD (at least the first pass I end up doing) meets the business needs it is very hard to get 100% coverage from a business scenario.

Finally I got it up to 99.84% coverage. This last standout was VERY annoying, specifically looking at the code coverage everything was green – no red or yellow sections. Hrrrmm. Ok it was a simple function that has this 1 block of unexecuted code – a switch statement taking a enum in as a parameter.  So lets have a look have a look (note I’ve stripped a lot of the content and comments out for readability:

   public enum PasswordStrengthIndex
   {
      None = 0,
      Weak = 1,
      Medium = 2,
      Strong = 3,
      MostStrong = 4,
   }

      public void ResetToDefinedPolicyStrength(PasswordStrengthIndex strength)
      {
         switch (strength)
         {
            case PasswordStrengthIndex.None:
               MinimumPasswordLength = 0;
               break;
            case PasswordStrengthIndex.Weak:
               MinimumPasswordLength = 4;
               break;
            case PasswordStrengthIndex.Medium:
               MinimumPasswordLength = 6;
               break;
            case PasswordStrengthIndex.Strong:
               MinimumPasswordLength = 8;
               break;
            case PasswordStrengthIndex.MostStrong:
               MinimumPasswordLength = 12;
               break;
         }
         return;
      }

The test case iterated through the enums and verified the output matched the expectations. Still I had this annoying block that was unexecuted. The next step was the Jimmy Neutron “Think Think Think”, a-ha! No default handler – so add one. Hmmm I cant add one that actually can be exercised as I’ve already used all my values in my enum. Well adding a default handler to anyone of those case statements still didnt resolve the issue. Now it was getting personal! I was dangerously close to calling in technical gurus and venting my disgust! So before making the calls time to search Google for “Code Coverage Switch” – and boy Google never ceases to amaze me.

Firstly it confirmed my suspicion that is was indeed the switch statement, second was a link that explained it in detail. Basically the default handler was not getting called – and there were a number of ‘breaking’ workarounds (change the enum to have another element to be used in the default case) in the Google answers, but one one that I ‘liked’ the most was to force the cast (which I would link out to the original author out of respect but I cant find it now ).

      public void ResetToDefinedPolicyStrength(PasswordStrengthIndex strength)
      {
         switch (strength)
         {
            case PasswordStrengthIndex.None:
               MinimumPasswordLength = 0;
               break;
            case PasswordStrengthIndex.Weak:
               MinimumPasswordLength = 4;
               break;
            case PasswordStrengthIndex.Medium:
               MinimumPasswordLength = 6;
               break;
            case PasswordStrengthIndex.Strong:
               MinimumPasswordLength = 8;
               break;
            case PasswordStrengthIndex.MostStrong:
               MinimumPasswordLength = 12;
               break;
            default:
               throw new ArgumentException("Supplied strength is not recognized as enum", "strength");
         }
         return;
      }

      [TestMethod()]
      [ExpectedException(typeof(ArgumentException))]
      public void ResetToDefinedPolicyStrengthBogus()
      {
         PasswordPolicy policy = new PasswordPolicy();
         policy.ResetToDefinedPolicyStrength(<strong>(PasswordStrengthIndex)666</strong>);
      }

Look at the last line of the listing above. Use a cast to force the enum to be a number out side of the range it was looking for – ta da!  On a partially related note this reminds me of the fact Enums are definitely interesting beasts that can bite you if you have compiled against one then change its definition in a different assembly.

So now we have got to 100% code coverage – woo hoo! The realization (which is obvious and well know to many) is that without TDD you would most likely end up writing tests that cover your code – but miss the business requirements. So any programmers out there who think they have good code coverage but are not using TDD are likely to have a false believe in their code because the tool says 100% green (unless they are coding guru’s!) – they are only really testing what they have written and not the business requirements. On the flipside just writing TDD tests without coverage is highly unlikely to meet all the ‘code’ edge cases (different than ‘business’ edge cases). So the conclusion it that the blend of the two are the best, but yet still not perfect. You have to start with TDD, and then move to code coverage – business first, code second.

I’m hoping at some point management teams will start to understand that even with 100% executed code coverage there can still be bugs as its a data world we live in! Obviously I’m an optimist!

Gareth

For starters I recommend reading the article [http://en.wikipedia.org/wiki/Password_strength]. This is a good article covering the relative strengths of passwords, and gives a guide for determining the strength of a random password and a human derived password.

The major problem with passwords are that humans need to remember them, or they write them down. In an interesting technology twist historically you only used to have to worry about your co-workers having access/abusing your password because there was implicit physical security in place – you could only log on if you were physically in the office.  As such at that time your biggest threat was your co-workers, unfortunately the secondary defense of physical location has effectively been removed with the internet and VPN technology.  So now your threat count has increased from the people you work with to the entire world! Add to this these people are financially motivated and can directly target you – its a whole lot scarier out there now!

So before jumping into the implementations we need to go through well known things to avoid to help improve password strength:

• Avoid sequences – keyboard or alphabet based (abcd, qwert, 1234, !@#$% etc)
• Avoid dictionary words, especially common ones! Be aware that common misspellings are also used in dictionary based attacks – so unless your misspelling is VERY unusual then you can expect it to be in a dictionary!
• Avoid leet/1337 password substitution of words (eg P@ssw0rd, M1cr0$0ft, 0\/\/n3d). Again these are now all in dictionaries, so while it may be harder to brute force – they are pretty trivial for a dictionary attack. Of course it doesnt hurt to be 1337, but it just really doesn’t help defend a targeted attack.
• Avoid team names, socials, license names etc.

Things to avoid to minimize compromise exposure:

• Use different passwords for different online accounts
• Avoid using information about you that can be readily be found on the web as a password reset scheme. DOB, where you were born, school name etc.
• If any account needs the most rigorous password control it is your email account. Nearly every online system ties back to an email account. If you need to reset a password, it normally goes to your email address. If that is compromised then that is really the opening of Pandora’s box.

Alright, lets start with the weakest ‘safe’ approach – Information entropy:

• This strength calculation only holds true for ‘random’ passwords. No human (at least that I know) can really generate a random password on their own. The best approach that I’m aware of is to start up notepad and get your two year old to start smacking your keyboard. Then take this text and randomly change case of characters and inserting special characters. Unfortunately this is still weak because we have 2 hands and the keyboard is naturally divided into where your hands go. This generation is not as randomly distributed as people would think – nor would I recommend it! But at least you have a starting point, but then you have to write it down!
• [0-9] – 10 possible symbols per character – 3.32 bits of base2 log entropy
• [a-z] – 26 possible symbols per character- 4.7 bits of base2 log entropy
• [A-Z] – 26 possible symbols per character- 4.7 bits of base2 log entropy
• [A-Z, 0-9] – 36 possible symbols per character- 5.17 bits of base2 log entropy
• [A-Z,a-z] – 52 possible symbols per character- 5.7 bits of base2 log entropy
• [A-Z, a-z, 0-9] – 62 possible symbols per character- 5.95 bits of base2 log entropy
• [A-Z, a-z, 0-9, Special] – 94 possible symbols per character – 6.55 bits of base2 log entropy

So we can see that having a strong password using completely random information will be hard to generate on our own, yet this approach is what is what is most commonly used to in web applications to determine password strength. This is not strong enough because humans are naturally not random. Using this theory the following non-random passwords generate results that imply the passwords are strong:

• 12345678901234567890 – 20*3.32 => 66.4 bits of entropy
• !!!!!!!!!!!!!!!!!!!! – 10 * 6.55 => 65.5 bits of entropy
• !@#$%^&*() – 10 & 6.55 => 65.5 bits of entropy
• qwertyuiop[]qwertyuiop[] = 24 * 6.55 => 157.2 bits of entropy.

The more astute among us will see the last two passwords were generated by running your finger across the a keyboard line of on a US keyboard. To enter the 24 characters password took under 3 seconds. So if anyone saw someone entering a password like this at work or in a library – its pretty easy to duplicate. Plainly you can see that with human users they are going to opt for the easiest way to remember and enter a password – this will never be random!

So to help avoid our users from becoming victims we have to try to take away the ‘easy’ passage from them. We have to assume the password is not going to be mathematically random – so we need to start from a different position. We have to ensure we remove the human weaknesses that other ‘black hats’ are looking to exploit.

So going back to the beginning of the article we are going to create an interface to define a ‘password policy’ that provides us a way to help enforce a stronger passwords – or  at least allows systems to setup a common language for handling passwords.

   /// <summary>
   /// Interface for defining a password policy
   /// </summary>
   /// <remarks>
   /// This security policy determines whether passwords
   /// meet pre-determined complexity requirements.
   ///
   /// If this policy is enabled, passwords must meet the
   /// following minimum requirements:
   ///
   /// Not contain the user's account name or parts of the
   /// user's full name that exceed four consecutive
   /// characters.
   /// Be at least <see cref="MinimumPasswordLength"/>
   /// characters in length
   /// Contain characters from three of the following
   /// four categories:
   /// English uppercase characters (A through Z)
   /// English lowercase characters (a through z)
   /// Base 10 digits (0 through 9)
   /// Non-alphabetic characters (for example, !, $, #, %)
   ///
   /// Complexity requirements are enforced when passwords
   /// are changed or created.
   /// </remarks>
   public interface IPasswordPolicy : IPolicy
   {
      /// <summary>
      /// Indicates the minimum password strength index for
      /// this policy (see PasswordStrengthIndex)
      /// </summary>
      /// <remarks>
      /// This value is based of a calculation of
      /// information entropy after sequences
      /// and dictionary words have been
      /// removed.
      /// </remarks>
      /// <value>
      /// The minimum index of the password strength.
      /// </value>
      PasswordStrengthIndex MinimumPasswordStrengthIndex
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the minimum length of the password.
      /// </summary>
      /// <value>The minimum length of the password.</value>
      int MinimumPasswordLength
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the maximum length of the password.
      /// </summary>
      /// <value>The maximum length of the password.</value>
      int MaximumPasswordLength
      {
         get;
         set;
      }

      /// <summary>
      /// If policy requires mixed case
      /// </summary>
      /// <value>true if policy needs mixed case</value>
      bool RequireMixedCase
      {
         get;
         set;
      }

      /// <summary>
      /// If policy needs digits
      /// </summary>
      /// <value>true if policy needs digits.</value>
      bool RequireDigits
      {
         get;
         set;
      }

      /// <summary>
      /// If policy needs special characters
      /// </summary>
      /// <value>
      /// true if require special characters are needed
      /// </value>
      bool RequireSpecialCharacters
      {
         get;
         set;
      }

      /// <summary>
      /// Indicates if the username needs to be additionally
      /// supplied to verify the password complexity against
      /// </summary>
      /// <value>
      /// true require username to check password against
      /// </value>
      bool RequireUsernameToCheckPasswordAgainst
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the maximum count of characters
      /// in a sequence
      /// </summary>
      /// <value>The maximum count of characters
      /// in a sequence.</value>
      int MaximumCharacterSequenceCount
      {
         get;
         set;
      }

      /// <summary>
      /// The duration of the lockout in minutes.
      /// </summary>
      /// <remarks>
      /// This security setting determines the number of
      /// minutes a locked-out account remains locked
      /// out before automatically becoming unlocked.
      /// The available range is from 0 minutes through
      /// 99,999 minutes.
      /// If you set the account lockout duration to less
      /// than zero, the account will be locked out until an
      /// administrator explicitly unlocks it. If an account
      /// lockout threshold is defined, the account lockout
      /// duration must be greater than or equal to
      /// the reset time.
      /// </remarks>
      /// <value>The duration of the lockout.</value>
      int LockoutDuration
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the lockout threshold.
      /// </summary>
      /// <remarks>
      /// This security setting determines the number of
      /// failed logon attempts that causes a user
      /// account to be locked out. A locked-out
      /// account cannot be used until it is reset
      /// by an administrator or until the
      /// lockout duration for the account has expired. You
      /// can set a value between 0 and 999 failed
      /// logon attempts. If you set the value to 0,
      /// the account will never be locked out.
      /// </remarks>
      /// <value>The lockout threshold.</value>
      int LockoutThreshold
      {
         get;
         set;
      }

      /// <summary>
      /// Reset account lockout after X minutes
      /// </summary>
      /// <remarks>
      /// This security setting determines the number of
      /// minutes that must elapse after a failed logon
      /// attempt before the failed logon attempt
      /// counter is reset to 0 bad logon attempts.
      /// The available range is 1 minute to
      /// 99,999 minutes.
      /// </remarks>
      /// <value>The duration of the lockout.</value>
      int LockoutResetInMinutes
      {
         get;
         set;
      }
   }

You can see this password policy template extends the initial outline to not only provide guidance for the number of entropy bits, but allows for the policy to cover the lock out strategy in the case of incorrect password handling and password expiry approaches. If you look at the source code you will also see the options that are available, but for the sake of this article we are trying to keep on point

So on to the actual strength testing, this oddly is rather simple at the end of the day. We are going to use an Interface definition (IPassword) for the Password processor (makes testing & mocking easier) so we can actually have multiple implementations (think MEF!).  Now the actual implementation.

1. Check for sequences using various lookup tables to determine if any sequences exist. If a sequence length is detected, and is longer than allowed the password fails the policy. The tables include:
   • Alphabetic + numeric sequence
   • QWERTY US Keyboard
   • QWERTY UK Keyboard
   • AZERTY Keyboard
2. Perform simple DecodeEliteEncoding then perform a simple hardcoded dictionary match of well known super common passwords
3. If supplied (and if required) compare password elements to the user name

The end implementation is still fairly simple and it would be fairly easy to improve on this implementation.  The most obvious ones are to support a custom dictionary and add more custom keyboard sequences. Other extensions would be to store the passwords and become a real password token service. We can leave it up to the reader to provide an implementation of IPassword to call the Google password rating service rather than the above implementation:

https://www.google.com/accounts/RatePassword?Passwd=csharphacker

All good stuff! I hope this helps (and the source code) people provide a better approach to helping strengthen passwords.

[Download source code here]

The linked source code is liable to change over time so check back often. The source code uses the Microsoft testing framework and currently has 100% code coverage! Although I don’t think 100% is all that people think it is.

Finally the goal is to make everything a more secure place – and in reality the best approach is to use a strong memorable password in conjunction with a hardware token that changes every minute.

As always feedback is welcome!

Gareth

• [The Virtualization Nation Podcast – Episode 3: Want to boot a physical computer from a VHD?]
• [Step-By-Step: Turning a Windows 7 DVD or ISO into a Bootable VHD Virtual Machine]

Seriously cool – I mean seriously! Native booting to a VHD on your disk – I want to buy who ever thought of that and got management to agree to putting that into Windows 7 (think HyperVisor) a beer!

I LOVE IT!!

Firstly it should be said that the standard GZipStream stream provides the functionality I’m sure the MS engineers expected it to do, which was for HTTP based compression (at least I think that was its expected purpose). However it is certainly not a fully featured class that is really easy to use for the programmers looking to get quick & helpful access to the GZip compression.

Specifically the problem I needed to solved was I needed to know how big any given “.GZ” decompressed file was without fully reading and decompressing the file. It seemed trivial enough – “gzip.exe -l” does what I needed, but no amount of hunting within MSDN helped. So on to the ever handy GZip wikipedia entry that detailed enough of the file format and provided the reference to the “GZIP file format specification version 4.3“.

So armed this this information we can start to decode the GZip file format to extract the length. Infact this class will check the file to see if it is GZip compressed and returns the decompressed length for that or the regular file length if it is not compressed.

The following class functions have been implemented (see the bottom of the article for the link to the full project):

   /// <summary>
   /// Utility class to help with managing GZip (.gz) files in .Net
   /// </summary>
   /// <remarks>
   /// This is a trivial wrapper class on top of <see cref="GZipStream"/> that does a little magic
   /// under the covers by looking at the underlying data format and retrieves the
   /// stored data information within the GZip compressed file.
   /// </remarks>
   public class GZipHelper
   {
      /// <summary>
      /// Gets the compressed file details
      /// </summary>
      /// <param name="filename">The filename.</param>
      /// <returns>True if file exists, else false</returns>
      public bool GetFileDetails(string filename);

      /// <summary>
      /// Gets the compressed file information from a file stream
      /// </summary>
      /// <param name="fileStream">The file stream.</param>
      /// <remarks>
      /// Definitions provided by RFC 1952 -GZIP File Format Specification (May 1996).
      /// Coding was performed against ftp://ftp.isi.edu/in-notes/rfc1952.txt
      /// </remarks>
      public void GetFileInformation(FileStream fileStream);

      /// <summary>
      /// Compresses the file file
      /// </summary>
      /// <param name="filename">The filename.</param>
      /// <param name="overWriteExisting">if set to <c>true</c> [over write existing].</param>
      /// <returns></returns>
      public void CompressFile(string filename, bool overWriteExisting);

      /// <summary>
      /// Decompresses the file.
      /// </summary>
      /// <param name="filename">The filename.</param>
      /// <param name="overWriteExisting">if set to <c>true</c> [over write existing].</param>
      /// <returns></returns>
      public bool DecompressFile(string filename, bool overWriteExisting);

      /// <summary>
      /// Returns a seekable stream into either a file or compressed file (defaults read-only)
      /// </summary>
      /// <remarks>
      /// Decompresses the stream into a <see cref="MemoryStream"/> if the file is compressed
      /// otherwise just returns back a regular <see cref="FileStream"/> as a <see cref="Stream"/>
      /// </remarks>
      /// <param name="filename">The filename to open.</param>
      /// <returns>Reference to opened stream</returns>
      public Stream GetSeekableStream(string filename);
   }

In combination to this the following properties are available:

• CompressedLength – Size of the compressed file (or regular file size if not compressed)
• DecompressedLength – Size of the file if it were uncompressed (or regular file size if not compressed)
• IsTextFile – Indicates if GZip thought the file was text based, potentially leading to better compression
• CompressionModeValue – Numeric indication of the compression mode used
• CRC16Present – Indicates a CRC16 is available for the file
• ExtraFieldsPresent – Additional meta fields are available in the file
• FileNamePresent – GZip contains the original file name
• FileCommentPresent – Compressed file has a comment associated with it
• IsCompressed – Indicates if the file is GZip compressed or not
• CompressedDate – If stored this is the date the file was compressed.
• CRC32 – CRC32 value associated with the file

Along with the project there are MSTest harnesses to test the class (trivial implementations). So the features of the class are:

• Can trivially determine a true file size (regardless if it was compressed via GZip or is uncompressed). This makes your code path much more readable if you are dealing with mixed file types.
• Provides a Seekable stream into the compressed file via via a MemoryStream. The key is that you dont need to worry about the compression (unless you are reading in BIG files) as you will get back a Stream for either a File or a Compressed file – both support seeking. This can be handy if you problem assumes it can Seek in the stream and you need to access GZip files!
• Trivial Decompress file, this also honors the CompressedDate. If that date is set then the decompressed file has that creation date.
• Trivial Compress file. Unfortunately at the time of writing I’ve not updated the header to include the date of the compressed file. This may come in a later version (and if so I’ll update the blog – but definitely no promises!).

Simple example usages are (taken straight from the unit tests!):

// Perform a file compression
GZipHelper actual = new GZipHelper();
actual.CompressFile(_fileName, true);

// Perform a file decompression
GZipHelper actual = new GZipHelper();
string fileName = "CSharpHackerSmallTest.txt.gz";
actual.DecompressFile(fileName, true);

// Get a seekable stream
GZipHelper actual = new GZipHelper();
using (Stream dataStream = actual.GetSeekableStream("CSharpHackerSmallTest.txt.gz"))
{
    // Silly seek - but it just shows it can be done
    dataStream.Seek(0, SeekOrigin.Begin);
    StreamReader sr = new StreamReader(dataStream);
    string contents = sr.ReadToEnd();

    Assert.AreEqual(119, contents.Length);
}

// Gets natural decompressed file length from a compressed file.
GZipHelper actual = new GZipHelper();
actual.GetFileInformation("CSharpHackerSmallTest.txt.gz");
Assert.AreEqual(119, actual.DecompressedLength);

Finally it should be noted that by all accounts the standard implementation of GZipStream in the base .Net libraries (actually the DeflateStream) has a problem when attempting to compress random or already compressed data. There is a Microsoft Connect article [http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=93930] that details the issue.

The GZipStream and DeflateStream classes can _significantly_ increase the size of “compressed” data. That means, they don’t just add a few header bytes as stand-alone compressors do, but they _inflate_ the data by as much as 50%. This is apparently because these classes do not check for incompressible data which is a standard feature of all stand-alone compressors. Both classes work fine when the data actually can be compressed.

Please refer to this thread for more details:

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=179704&SiteID=1

The base implementation worked for me and met my specific needs without the need of bringing in any third party DLLs. Which incidentally also has a nice benefit for those looking to bring this into proprietary software of avoiding any licensing discussions with supervisors! If you want a more robust GZipStream implementation you can check out http://dotnetzip.codeplex.com/. This apparently has a drop in replacement, but this class could still be useful even if use this drop in replacement as well.

I hope this helps some one out there

[Download GZipHelper (Source + Project) Here]

This download link will always have the latest and greatest version.

Gareth

[A primer on setting base-addresses for managed DLLs] covers how for performance reasons you probably want to [NGen] your code (on the remote computer) and ensure you have your base-addresses setup to avoid clashes. The interesting thing is that with [ASLR] enabled on VISTA, Windows 7, Windows Server 2008 and higher it doesnt help .

So for the performance conscious .net coders out there targetting fast startups definitely have a read.

Gareth

When I was doing some unit tests I got this message

Failed to queue test run ‘Gareth@Monster 2009-07-19 21:41:43′: Test Run deployment issue: The location of the file or directory ‘d:\home\git\filehash\filehashtest\bin\debug\System.Data.SQLite.dll’ is not trusted.

This has all to do with me trying to be a good boy and downloading the latest DLL from web to ensure I was developing and testing on the absolute latest version. This helpful ‘magic’ is the Microsoft “Attachment Manager” that was introduced in XP SP2. Now it seems that both Firefox and WinZip 10 honor this feature (http://www.winzip.com/xattmgr.htm) so if you unzip a file using Winzip that has been tagged as ‘from the internet’ the unzip then tags all the files coming out as ‘blocked’.

There are a varying number of solutions:

Option 1 – Pinpoint accuracy

Select the properties on the file in question and press “Unblock”.

Attachment Manager Blocking In Action

Good if you only have 1 file to deal with, bad if you have a large number of them! Unfortunately there is no good way to do this via a multi-select option, but remember if they all came from a ZIP file un ‘block’ the ZIP before performing the UNZIP!

Option 2 – ‘The ultimate sledge hammer’ – definitely for the brave

I dont recommend this!! However if you are feeling brave (and this is a personal PC and not a corporate one!) this feature can be disabled using search for “Do not preserve zone information in file attachments” in the link http://support.microsoft.com/default.aspx/kb/883260 – its a ways down!

This won’t remove alternate streams from existing files. Also this ‘might’ have side effects so use with caution.

In theory the Attachment Manager can be disabled via gpedit.msc. I say in theory because I cant see the option I’m going to write about (http://www.lansa.com/support/notes/p0348.htm)

The steps to do that are outlined below:

1. Run the ‘Group Policy Object Editor’ (gpedit.msc) either from the Command Prompt or using the ‘Run’ command in the Windows Start Menu
2. Expand the node ‘User Configuration’ >> ‘Administrative Templates’ >> ‘Windows Components’ >> ‘Attachment Manager’.
3. Right-click on the third option: Do not preserve zone information in file attachments. Choose ‘Properties’.

Option 3 – Using Streams (for the script folks – and still a little brave!)

Download the handy dandy ‘streams’ from http://technet.microsoft.com/en-us/sysinternals/bb897440.aspx. Then run “Streams -d –s c:\yourdirectorywithblockedfiles” this will recurse through all the folders deleting any streams it comes across. Beware some programs may be using streams for storage, if they are you will be deleting that meta data. But if this is only in a development folder you should be OK (make sure you have backups!)

Option 4 – Copy all the affected files to a FAT file system and copy them back

Title says it all, since streams are not supported on FAT file systems performing this copy to a FAT system and then back to NTFS ‘removes’ the offending streams.

Good luck

So I’m finally taking the plunge as PowerShell 2.0 is about to come out. Firstly I’ll cover why I’m now making the move rather than before, then will provide some good links to help get your skills up!

Why update now?

Well Powershell has been out for a long time now, but the big differentiator is that it is now coming part of the standard Windows OS release (Windows 2008, Windows 7) rather than being an additional download. For any developer that has to support 3rd party systems (aka systems you dont own) you ideally want to stick to the standard tools installed on the machine rather than having to get a number of pre-requisites.  As it is only installed by default on a small section of machines its far from a slam dunk – but once it gets into the OS distribution we can be fairly sure its going to be mainstream and remain supported!

Memory footprint

So on to most commonly cited downside made by memory conscious folks, so lets get that out of the way!

Randomly Performed on my Windows XP (SP3) – these are definitely not exact but should show orders of magnitude.

• Command – Cmd.exe – Mem Usage 2,572Kb, VMSize 2,008Kb
• TCC LE – TCC.exe – Mem Usage 3400Kb, VMSize 2,944Kb
• Powershell 1.0 – Powershell.exe – Mem Usage 25,324, VMSize 22,668Kb

So obviously it is definitely an order of magnitude heavier than historic command shell implementations. There are several reasons for this, the biggest is that the shell uses a plug-in architecture so you can seamlessly add your own ‘cmdlets’ into the shell. Understandably if you start to run cmdlets (and most everything is a cmdlet) your memory foot print starts to go up as they get loaded into the process. For example if you run “PS”  to get a list of running processes your memory footprint will change as that module is loaded and executed. So if you are only concerned about memory footprints you should stop reading now – but if you are looking for functionality this is where PowerShell really shines!

Standard Cmd.exe like Navigation

After starting up powershell (“Start/Run/Powershell”) you start with a command like prompt. In my case its:

PS C:\Documents and Settings\csharphacker>

From here you can do the normal things:

• CD – Change Directory
• DIR – Directory listing
• REN – Rename
• DEL/ERASE – Delete
• PUSHD – Push current location to stack
• POPD – Pop last location from stack
• CLS – Clear Screen
• COPY – Copy file

Also notice there is tab completion of file names , now standardly available – but a killer not to have!

For Unix folks running on Windows there are Unix equivilents

• LS – Directory List
• RM – Remove file
• PWD – Current directory
• PS – Process list
• CAT – Output contents of file

Still pretty standard stuff. As you look closer you will see that infact these are not build-in commands, but actually aliases to other commands within the shell.

CommandType     Name                            Definition
-----------     ----                            ----------
Alias           ac                              Add-Content
Alias           asnp                            Add-PSSnapin
Alias           clc                             Clear-Content
Alias           cli                             Clear-Item
Alias           clp                             Clear-ItemProperty
Alias           clv                             Clear-Variable
Alias           cpi                             Copy-Item
Alias           cpp                             Copy-ItemProperty
Alias           cvpa                            Convert-Path
Alias           diff                            Compare-Object
Alias           epal                            Export-Alias
Alias           epcsv                           Export-Csv
Alias           fc                              Format-Custom
Alias           fl                              Format-List
Alias           foreach                         ForEach-Object
Alias           %                               ForEach-Object
Alias           ft                              Format-Table
Alias           fw                              Format-Wide
Alias           gal                             Get-Alias
Alias           gc                              Get-Content
Alias           gci                             Get-ChildItem
Alias           gcm                             Get-Command
Alias           gdr                             Get-PSDrive
Alias           ghy                             Get-History
Alias           gi                              Get-Item
Alias           gl                              Get-Location
Alias           gm                              Get-Member
Alias           gp                              Get-ItemProperty
Alias           gps                             Get-Process
Alias           group                           Group-Object
Alias           gsv                             Get-Service
Alias           gsnp                            Get-PSSnapin
Alias           gu                              Get-Unique
Alias           gv                              Get-Variable
Alias           gwmi                            Get-WmiObject
Alias           iex                             Invoke-Expression
Alias           ihy                             Invoke-History
Alias           ii                              Invoke-Item
Alias           ipal                            Import-Alias
Alias           ipcsv                           Import-Csv
Alias           mi                              Move-Item
Alias           mp                              Move-ItemProperty
Alias           nal                             New-Alias
Alias           ndr                             New-PSDrive
Alias           ni                              New-Item
Alias           nv                              New-Variable
Alias           oh                              Out-Host
Alias           rdr                             Remove-PSDrive
Alias           ri                              Remove-Item
Alias           rni                             Rename-Item
Alias           rnp                             Rename-ItemProperty
Alias           rp                              Remove-ItemProperty
Alias           rsnp                            Remove-PSSnapin
Alias           rv                              Remove-Variable
Alias           rvpa                            Resolve-Path
Alias           sal                             Set-Alias
Alias           sasv                            Start-Service
Alias           sc                              Set-Content
Alias           select                          Select-Object
Alias           si                              Set-Item
Alias           sl                              Set-Location
Alias           sleep                           Start-Sleep
Alias           sort                            Sort-Object
Alias           sp                              Set-ItemProperty
Alias           spps                            Stop-Process
Alias           spsv                            Stop-Service
Alias           sv                              Set-Variable
Alias           tee                             Tee-Object
Alias           where                           Where-Object
Alias           ?                               Where-Object
Alias           write                           Write-Output
Alias           cat                             Get-Content
Alias           cd                              Set-Location
Alias           clear                           Clear-Host
Alias           cp                              Copy-Item
Alias           h                               Get-History
Alias           history                         Get-History
Alias           kill                            Stop-Process
Alias           lp                              Out-Printer
Alias           ls                              Get-ChildItem
Alias           mount                           New-PSDrive
Alias           mv                              Move-Item
Alias           popd                            Pop-Location
Alias           ps                              Get-Process
Alias           pushd                           Push-Location
Alias           pwd                             Get-Location
Alias           r                               Invoke-History
Alias           rm                              Remove-Item
Alias           rmdir                           Remove-Item
Alias           echo                            Write-Output
Alias           cls                             Clear-Host
Alias           chdir                           Set-Location
Alias           copy                            Copy-Item
Alias           del                             Remove-Item
Alias           dir                             Get-ChildItem
Alias           erase                           Remove-Item
Alias           move                            Move-Item
Alias           rd                              Remove-Item
Alias           ren                             Rename-Item
Alias           set                             Set-Variable
Alias           type                            Get-Content

So if you start to look at this list you will begin to understand why this is a bigger beast than CMD or TCC LE will ever be.  However it has to be said that these commands are not “identical” to your preferred flavor – but more of a likeness to it. So LS doesnt have exactly the same characteristics, the same was that DIR doesn’t either – but the ‘intent is the same’.

The best keyboard shortcut for me is F8 (but I want to remap it to the up arrow!)  – read more about shortcuts can be found here.

Recommended Reading Links (I should point out I am not affiliated with any of the below, but they are what I have used to get more acquainted with PowerShell) :

• Microsoft Powershell Owners Manual
  • Great starting place to this scripting beastie! Have a look at the chapters here – excellent material.
• Scripting with Windows PowerShell
  • Highly recommended as a starting place on PowerShell
• Free PowerShell book “Mastering PowerShell”
  • 500+ pages of well written handy examples

After reading these you should have a better understanding of the power of power shell. If using PowerShell to read/manipulate the registry, open CSV files, perform credential maintenance via script still doesnt make you want to learn it then remember that SQL 2008 has Powershell integration. So now you can browse your SQL schema using PowerShell as well.

This is where we can say for a certainty “Microsoft has started to take scripting seriously”!

Pretty cool stuff,
Gareth

The official information can be found below:

• http://blogs.msdn.com/windowsazure/
• http://blogs.msdn.com/windowsazure/archive/2009/07/14/confirming-commercial-availability-and-announcing-business-model.aspx

Fundamentally the problem for me is that it is competing head to head with Amazon with no clear winner. Infact Amazon not only has a small price advantage with their Linux offering, but they have it all running with lots of infrastructure setup over many years. They are the defacto ‘standard’ and they are cheaper on Linux and only a tiny bit more expensive on Windows!

Unfortunately at first blush this doesnt seem to be a business compelling reason to switch to Azure, and evaluation which one are you going to choose? A company that has been doing it for years and has been maturing their model and offerings or the new one to the block. I’m sure Microsoft will change their pricing model – but it appears to be a surprisingly weak opening offer from Microsoft.

A new news/blog articles covering this can be found here:

• Microsoft’s Azure cloud price pipped by Amazon’s Linux
• Azure Announces Unimaginative Pricing Model

Thoughts? Am I missing something fundamental here?

Gareth

• AWS Start-Up Challenge For 2009 – Amazon has kicked off the third annual AWS Start-Up Challenge now.
  • Start-ups in the United States, the United Kingdom, Germany, and Israel are encouraged to apply for a chance to win $50,000 in cash, $50,000 in AWS credits, mentoring sessions from AWS technical experts, and AWS Premium Support Gold for one year.
• How to use a CTE in a view. If you dont know what a CTE is check it out here, definitely check out its recursive capabilities here.
• Microsoft Research “Gazelle” fires experimental salvo at Google.
• PCI DSS Legitimizes Conflicts of Interest
• Geneva identity grows up with rebranding roll on Active Directory Federation Services (ADFS), Windows Identity Foundation & Windows CardSpace.
• Managed Extensibility Framework (MEF) Preview 6: V1 Feature Complete Silverlight Support and Much More!
  • Not only is this the feature complete build for MEF V1.0 (which will ship with .NET Framework 4) but it also has the first drop of MEF for Silverlight!
• Google releases remote screen viewer NeatX.
  • “The good old X Window system can be used over the network, but it has issues with network latency and bandwidth. Neatx remedies some of these issues,” Google engineers wrote on the company’s open source blog.

• MD5 - Dont use if you can avoid it as this is known to have vulnerabilities and should never be used!
• RIPEMD160 – This is supported by .Net, but isnt really heavily used. Recommend using SHA256 or SHA512
• SHA1- If you can avoid it use SHA256 or SHA512
• SHA2 Family
  • SHA256
  • SHA384
  • SHA512

If you haven’t come across MD5Sum.exe, SHA1Sum.exe, SHA256Sum.exe you can find native Windows ports here (or if you are looking for the more official GNU versions they can be found here) . Which if you are just looking for the command line tools that is probably enough. However sometimes you may have the need to do all this work yourselves in C#, if so this is the article that should help guide you!

First of all this is going to be fairly simple as the .Net library supports all of the above hash formats, so all we are really talking about doing is showing you the best way to use the supplied .Net runtimes to perform your hashing. So on to the magic (note to avoid width formatting issues this isnt exactly how I normally format the code!):

/// <summary>
/// Performs the SHA1 Hash function on file
/// </summary>
/// <param name="filename">
/// The filename to be hashed.
/// </param>
/// <returns>
/// SHA1 Hash value associated with the file
/// </returns>
public static string SHA1HashFile(string filename)
{
   string hashedValue = string.Empty;

   //create our SHA1 provider
   SHA1CryptoServiceProvider hashAlgorithm = new SHA1CryptoServiceProvider();

   //hash the data from the file
   byte[] hashedData = hashAlgorithm.ComputeHash(File.ReadAllBytes(filename));

   //loop through each byte in the returned byte array to convert into printed ASCII
   foreach (byte b in hashedData)
   {
      hashedValue += String.Format("{0,2:x2}", b);
   }

   //return the hashed value to the caller
   return hashedValue;
}

This does the SHA1 hashing of the supplied file – and matches the output of GNU version of SHA1Sum.exe. I told you it was simple . Dissecting this code should be pretty trivial:

1. Create the SHA1 Service provider (System.Security.Cryptography.SHA1CryptoServiceProvider)
2. Call ComputeHash passing in a byte array.
3. Take the results and output it as text. In case it wasnt obvious the results of the hash is a binary blob, hence the need to format it into a string friendly representation.

However there are really 2 problems with this code:

1. File.ReadAllBytes – This returns a byte array of the file – pretty much as you would expect! The problem is that if this is a very big file, for example a hash for a DVD, the entire file needs to be loaded into memory before it gets hashed. Obviously not the most optimal approach!
2. This is completely locked into SHA1, you need a new function for any other different hashing function. Not a biggie, but it would be nice to  get some reuse in now and then. Definitely useful if you see the full example where we have to use some fall back processing if an algorithm is not available.

Thankfully fixing this is still pretty trivial. To fix issue 1 rather than using the ComputeHash that takes in the byte array, use the one that takes the stream. This avoids the need for having the entire file in memory before the hash process can start. Out of curiosity I looked up the publicly available source code for the function to check it was in fact doing what I thought. Thankfully it is simple and obvious:

...
// Default the buffer size to 4K.
byte[] buffer = new byte[4096];
int bytesRead;
do {
   bytesRead = inputStream.Read(buffer, 0, 4096);
   if (bytesRead > 0) {
     HashCore(buffer, 0, bytesRead);
  }
} while (bytesRead > 0);
...

So we can see when we use the stream version of HashAlgorithm.ComputeHash Method (Stream) it only will use up a small memory chunk for calculating the hash values. So we are safe from big files from potentially killing the application.

Issue 2 – The .Net team did a nice job of creating base classes, one of which is HashAlgorithm. This is actually the class that implements the hashing ‘interface’. All hash algorithms must derive from this class. So we can use this to our advantage:

/// <summary>
/// Performs a Hash operation on the supplied file.
/// </summary>
/// <param name="filename">
/// The filename to be hashed.
/// </param>
/// <returns>
/// Selected Hash value associated with the file
/// </returns>
public static string HashFile(
          string filename
          , HashAlgorithm hashAlgorithm)
{
   if (!File.Exists(filename))
   {
      throw new ArgumentException(filename + " must exist", "filename");
   }

   string hashedValue = string.Empty;
   byte[] hashedData = null;

   // Create the stream
   using (FileStream fs = File.Open(filename, FileMode.Open, FileAccess.Read))
   {
      hashedData = hashAlgorithm.ComputeHash(fs);
   }

   //loop through each byte in the returned byte array
   foreach (byte b in hashedData)
   {
      //convert each byte and append
      hashedValue += String.Format("{0,2:x2}", b);
   }

   //return the hashed value
   return hashedValue;
}

Now we have a generic function to return back a hash value from any supported .Net hash algorithm. To call it you could just do “HashFile(filename,new SHA1CryptoServiceProvider())”. Voila! Performance and can trivially support any hashing class the .Net framework implements.

Ok so lets get a little more adventurous now. Attached to this entry is a very simple (aka not fully featured) HashSum source code that allows the same executable to be used to provide all the above hashing! However as a word of caution you need to be a little careful with the more advanced hashing. For example Microsoft supply both “SHA256CryptoServiceProvider” and “SHA256Managed”. On the surface they look pretty much the same, apart from SHA256CryptoServiceProvider is only available in .Net 3.5 (or higher). However since this uses Operating system cryptographic service providers they may not be available on the platform your program is running on. If that is the case then you will get the PlatformNotSupportedException exception thrown:

System.PlatformNotSupportedException was unhandled
  Message="The specified cryptographic algorithm is not supported on this platform."
  Source="System.Core"
  StackTrace:
       at System.Security.Cryptography.CapiNative.AcquireCsp(String keyContainer, String providerName, ProviderType providerType, CryptAcquireContextFlags flags, Boolean throwPlatformException)
       at System.Security.Cryptography.CapiHashAlgorithm..ctor(String provider, ProviderType providerType, AlgorithmId algorithm)
       at System.Security.Cryptography.SHA256CryptoServiceProvider..ctor()
       at CSharpHacker.Hash.HashSum.Main(String[] args) in X:\GIT\FileHash\FileSum.cs:line 76
       at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException:

Hmmm – you don’t read about that little chestnut in the MSDN help! So if you know you are going to only be running this on a Windows 2008, Vista or Windows 7 or later you can just use the “SHA256CryptoServiceProvider” version. However if you may have to support systems such as XP (potentially Windows 2003 as well) you will have to use the Managed versions. The safest route would be to provide a graceful fall back mechanism (possibly with a warning) that the CSP version could not be used and using the managed code version instead. This provides the best of both worlds, if the platform supports the CSP version you can use that (which should give you a speed increase) or you use the managed solution.

case "SHA256SUM":
default:
   try
   {
      hashAlgorithm = new SHA256CryptoServiceProvider();
   }
   catch (PlatformNotSupportedException)
   {
      // Fall back to the managed version if the CSP
      // is not supported on this platform.
      hashAlgorithm = new SHA256Managed();
   }
   break;

This is a second benefit of using the “HashAlgorithm” approach, the underlying code responsible for the generic hashing function is that it doesnt need to know what version (or even algorithm) it is using.

You also have to bear in mind that if you are going to use “SHA256CryptoServiceProvider” (or equivalent) you have to be using .Net 3.5 or greater.

Click to download Sample CSharp FileSum project

This is a .Net 3.5 project that based off the executable file name it uses that algorithm. So if you rename FileSum.exe to “SHA512Sum.exe” it will perform the SHA512 hash on the input file, MD5Sum.exe MD5 hash, etc. If the name is not a recognized name it defaults to using the SHA256 algorithm. This is not designed to be a wholesale replacement for SHA256Sum etc, but more of a guide how to write a fully featured version. So things missing from this include (and are not limited to ):

• No support for wildcards (simple enough to add but not there)
• No support for checking files match the input file (‘-c or –check’)
• Only binary mode is supported, no support for ASCII/Text mode. No support for ‘-t’ or ‘–text’
• No support for standard input processing

Hope you found this useful,

Gareth

is not supported

• JQuery visualize – Trivially converts an HTML table into a graph using 1 line of code “$(‘table’).visualize();”. Definitely read this one, I’m not entirely sold on the single line approach – but bang for buck its definitely up there. As you would expect it has been tested under IE6, IE7, IE8, Firefox 2, Firefox 3.5, Safari 3 and 4, Opera 9 – so interoperability is good
• JQueryui.com – jQuery UI provides abstractions for low-level interaction and animation, advanced effects and high-level, themeable widgets, built on top of the jQuery JavaScript Library, that you can use to build highly interactive web applications. The thing I like most with the JavaScript tools is that you can view the demos right in your browser.  JQueryUI has the concept of  Theming which is nice for web developers looking to allow their users to change the look of the applications.

Pretty cool stuff,

Gareth

• Google declares total war on Microsoft
• Google’s vanity OS is Microsoft’s dream
• Will Google Chrome OS Be Any Different than LINUX, BEOS, Network Computer, etc.?
• Google declares total war on Microsoft, but the main battles are years away
• Google’s Chrome Operating System: A Revolution in the Making?

Actually this is a rare time I disagree with “TheRegister”. I think this will have a potentially profound affect on computing as we know it. Years ago we didn’t have infrastructure to support net computing, but now with the various cloud services and the fact everyone wants connectivity from anywhere  – now times are different. Obviously only time will tell which is more accurate (no one is ever 100% right ).

However as an application developer it makes me wonder where applications are going to end up. Actually I think I know, but I’m not sure corporations have really cottoned on to the significant change in technical directions. So let me ask a different question that will circle back around to the impact of Google OS! Everyone is looking into supporting low/no footprint client applications (RIA), and we have a varied number of choices:

• Citrix/XWindows/X11 – Remote access via terminal sessions
• Flash – The classic RIA within the browser
• SilverLight – Microsoft RIA platform within the browser
• Click-Once-Applications – .Net applications deployable from the browser
• Java FX – Java RIA platform within the browser
• Browser/JavaScript/HTML5 – the browser!

So it would appear the browser is going to be a fairly core piece of architecture that is going to have to exist for a number of years, the question is what is going to become dominate in it? What is your company betting on?

Lets put these choices in a different order of the directions the various supplier companies want us to go in:

• Sun – Java FX, multi platform
• Microsoft – Silverlight/.Net click-once, Windows + Mono
• Adobe – Flash, multi platform
• Google – Browser/JavaScript

Google is betting on making everything web based and running in the cloud. Now back to the developer question, a couple of years ago there were limited mobile device penetration into the general masses. Blackberries were used by corporate folks to get important emails and general access – others had pagers! Now a significant growing percentage of people have ‘smart phones’ that have ‘unlimited’ data plans.  As a developer 5 years ago who would have spoken about supporting Safari as a browser for your application? With the iPhone how many times a day do you think a business person is thinking – I wish I had this application on my iPhone.

If Google sticks to pure Javascript within the browser they will have coverage EVERYWHERE. Apple cant lock them out through the Apple store, Microsoft cant disable JavaScript support within their browser to cut out Google.  Stability issues cant be attributed to a third party add-in such as JavaFX, Silverlight or Flash which doesn’t work so well on that specific platform – the browser is responsible.

Google applications will be globally available to all netbooks, Google OS, Android phones, Windows phones, iPhones and what ever technology platform is next (as long as it has a browser capable of JavaScript). Add to this that HTML is getting smarter with HTML5, things are definitely coming into place for Google.

So back to the question – What impact to developers? My guess is that if you have tried in the past to generate a HTML application and you (or your company) now no longer entertain that idea as it was hard/complicated/problematic – you will be changing your mind within a couple of years. Naturally there will be ever evolving more helpful tool-sets to further help development, but I think the pure web apps are back in force and will be unstoppably dominant. The question is “Are you ready to accept that fact yet?“…

Time will tell , but I’m betting its going to be nearly pure Web client applications with lots of cloud magic pulling the strings. Google has been quietly moving this way for a while and the Google OS is the beginning of the public face of this change, and as a result this will be a massive change that development shops need to cater for.

Gareth

Its been a while watching the various incremental builds coming out for Wix 3, but now we are at the RTM the beginning for 3.0! Over this past weekend it has been marked RTM , the 3.0.5419 build of WiX v3.0 was declared the final production build. Read more about the various Wix announcements http://www.joyofsetup.com/2009/07/04/wix-v3-0-has-been-released/

The Wix team is now working on 3.5 – for Visual Studio 2010! Time to upgrade all those earlier versions of 3.0 to the 3.0.5419 build.

For those interested here are some links:

• Aaron Stebner’s WebLog announcement (actually this has the below links in it)
• WiX 3.0 Manual (note this is currently blank when I checked it)
• Download the RTM version http://wix.sourceforge.net/releases/3.0.5419.0/

Many congratulations to the team!

Gareth

The first thing to do is go and grab the latest source! The source code is currently hosted on SourceForge at: http://sourceforge.net/projects/sqlite-dotnet2/. If you go to this page you there are links (or there were at the time of writing!!) to “Download Now!” or “View all files“. If you are looking for the source you need to chose “View all Files”. From here you should choose the latest source package (currently “SQLite-1.0.63.0-source.zip”) and download and unzip to your drive. You should now have the latest and greatest ‘stable’ source code, if you need to pick up an in flight fix or need to most up to date you may have to use CVS to download the package. Unfortunately (albeit fortunate for this article) “SQLite-1.0.63.0-source.zip” contains an issue where the build didnt completely work straight out of the box for Win32 builds. I contacted the team through the developer forum (http://sqlite.phxsoftware.com/forums/) and the problem was resolved in the latest CVS code patch (Patchset 415). So given this we need to pull it from CVS rather than the prepackaged code. NOTE: Normally speaking you should only need (and I would recommend only) the packaged code rather than CVS. Only if you want to live on the bleeding edge, or have a real need to get the latest release should you really use the CVS version. If fact the team doesnt even recommend using the CVS version (http://sqlite.phxsoftware.com/forums/t/1814.aspx).

However to complete the full circle and want to go down the CVS route and you are not familiar with CVS you can find out more information here. Raw CVS executables for Windows can be found here, but I would recommend if you are going to make a habit out of it you get and use TortoiseCVS instead.

cvs -d:pserver:anonymous@sqlite-dotnet2.cvs.sourceforge.net:/cvsroot/sqlite-dotnet2 login
cvs -z3 -d:pserver:anonymous@sqlite-dotnet2.cvs.sourceforge.net:/cvsroot/sqlite-dotnet2 co -P SQLite.NET

[Note if the first line fails create an empty ".cvspass" file and retry it]

Now you should have the latest code straight from the CVS repository, or if you are more sensible the packaged ZIP file covered at the beginning of the article.

Now on to the building! In the top level folder there are 2 interesting/key files:

• readme.htm – This details the latest release notes associated with the provider code, and covers the differences between the various historical releases.
• SQLite.NET.sln – This is our well loved and trusted VS2008 solution file!

Starting the solution file the solution loads up and with the latest version I’m using8 projects are loaded:

• SQLite.Interop
• System.Data.SQLite – Compact
• System.Data.SQLite – Managed Only
• System.Data.SQLite – Netmodule
• System.Data.SQLite.Linq
• test
• testce
• testlinq

Out of the box you should be able to build the release and debug files. You will also notice slightly different icons associated with the projects “System.Data.SQLite – Compact” & “testce”. This is because these builds are targeting a mobile or Compact framework device.  They dont hurt to be there so feel free to ignore them as they wont be used unless you a compact/mobile build type.

Congratulations you should have successfully built the components and can verify your build by running the “test” application.

Now on to the more technical deep dive of how it works

Firstly lets be clear – this is certainly a non-trivial build process! This is merging C code & .Net code into the same binary DLL using netmodules. More on netmodules can be found at “How to: Build a Multifile Assembly“. Frankly I’m going to just dive into a little more detail here as this concept is key to the build.

Purely looking at the project in Visual Studio is a little misleading:

Hmmm – a “Windows Application” – really…. However looking at the “System.Data.SQLite – netmodule.csproj” in a text viewer you see something very different:

<OutputType>Module</OutputType>

Obviously this is very different than “<OutputType>Library</OutputType>”, “<OutputType>WinExe</OutputType>”, “<OutputType>Exe</OutputType>”. This is the netmodule magic we covered above – all this is largely hidden from the user if they just look at the VS interface. This magic is then referenced through the SQLite.Interop.vcproj though:

<Tool
Name="VCLinkerTool"
AdditionalDependencies="..\System.Data.SQLite\bin\System.Data.SQLite.netmodule"
OutputFile="../bin/System.Data.SQLite.DLL"
ModuleDefinitionFile="src\sqlite3.def"
EmbedManagedResourceFile=""
DelayLoadDLLs="advapi32.dll"
RandomizedBaseAddress="1"
DataExecutionPrevention="0"
ImportLibrary=""
TargetMachine="5"
KeyFile="..\System.Data.SQLite\System.Data.SQLite.snk"
CLRUnmanagedCodeCheck="true"
/>

The good news about this is that this is just a ‘dependency’ and is fully supported in the C/C++ interface in VS2008. A little more can be read about here “.netmodule Files as Linker Input”

Alright so we now have the background knowledge to understand how this build is actually being used. So now lets look at the what the build sets up for us (the below is really the Win32/x64 build):

• Debug
  • SQLite.Interop
  • System.Data.SQLite – Managed Only
  • System.Data.SQLite.Linq
  • test
  • testlinq
• Debug – Stock
  • System.Data.SQLite – Managed Only
  • System.Data.SQLite.Linq
  • test
• Release
  • SQLite.Interop
  • System.Data.SQLite – Netmodule
  • System.Data.SQLite.Linq
  • test
• Release – Stock
  • System.Data.SQLite – Managed Only
  • System.Data.SQLite.Linq
  • test

The project dependencies are:

• SQLite.InterOp -> System.Data.SQLite – netmodule
• System.Data.SQLite.Linq -> SQLite.InterOp
• Test -> SQLite.InterOp

You can see that only the “Release” build doesn’t include the “System.Data.SQLite – Managed Only” version. Any build other than this one generates a build that requires both “System.Data.SQLite” and “SQLite.Interop.DLL”. The “Managed Only” isn’t a pure managed only version of SQLite, it still uses the native SQL C runtime code – but it makes the call out to the Interop dll. The ‘netmodule’ version is used to embed in the ADO.NET C# classes into the ‘System.Data.SQLite’ DLL, all other builds of ‘System.Data.SQLite’ just reference the interop DLL.

Release Build

As mentioned before, in the release build SQLite.Interop is where all the magic happens in this build! This is the one that outputs bin\System.Data.SQLite.DLL (note in all other builds this generates ‘SQLite.Interop.dll’ or in the stock case ‘sqlite3.dll’). However since this is really compiling the standard C SQLite code there is a little more magic going on here. This project has the linker dependency on “..\System.Data.SQLite\bin\System.Data.SQLite.netmodule”. This is how we get the C# ADO.NET provider injected into the System.Data.SQLite.DLL through the netmodules. However this process has the side effect of not being able to debug into the provider . So for source code debugging use the debug build that references “Managed Only” rather than the netmodule.

Debug Build

From now on it is fairly standard, although I have seen an odd quirk is that if you attempt to debug into the regular “Debug Build” an error is through regarding the “‘OutputType’ failed”, if you use the “Debug – Stock” build this goes away as the ‘netmodule’ is not included in the build.

Happy compiling and debugging!

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

• Mono on PS3 making progress
• Canonical offers tech support for clouds – Ubuntu + Eucalyptus = homegrown EC2
• Handy listing: MSDN Magazine: CLR Inside Out
  • All the CLR news from MSDN magazine. Definitely good stuff here.
  • Memory Usage Auditing For .NET Applications – Who doesnt need more help monitoring their .Net applications!
• When in doubt reboot
  • I don’t feel so bad now when I read a Microsoft engineer admits sometimes this draconian measure is sometimes needed. I would also read this is you have ever faced “Kerberos” ‘opportunities’.  It also mentions how they purged all of the Kerberos Tickets using KerbTray which is part of the Windows Resource Kit.
• GChart 2.5 released – Uses GWT for those using that.

Gareth

Most Simple – Data Source
Format: “Data Source=[DBFileName]”
Example: @”Data Source=C:\CSharpHackerDemo.db3″

This connection string will create the database if it doesnt already exist without notifying you. For some utility functions this may be exactly what is desired, in some cases you want to know your DB doesnt exist!

If you want to programatically create the file intentionally, presumably if  “!File.Exists()” you can do:

SQLiteConnection.CreateFile(@"C:\CSharpHackerDemo.db3");

Only Open if it exists – FailIfMissing
Format: “Data Source=[DBFileName];FailIfMissing=True;”
Example: @”Data Source=C:\CSharpHackerDemo.db3;FailIfMissing=True;”

With this connection string if the DB file is not present the open will fail.

Secure/Encrypt the DB – Password
Format: “Data Source=[DBFileName];Password=[123456789];”
Example: @”Data Source=C:\CSharpHackerDemo.db3;Password=[123456789];”

With this connection string if the DB file will be created or opened using the password as the encryption key.

Other options:

• ReadOnly – When enabled, the database will be opened for read-only access and writing will be disabled. Default False.
• CacheSize – Sets the cache size for the connection. Default value is 2000 bytes. Under the covers this calls “PRAGMA cache_size={0}”.
• PageSize – Sets the database page size. Under the covers this calls “PRAGMA page_size={0}”. Default is 1024 bytes.
• MaxPageCount – Sets the maximum number of pages the database may hold. Default is 0 which is unrestricted. Under the covers this calls “PRAGMA max_page_count={0}”.
• DateTimeFormat - Sets the datetime format for the connection. Default is ISO8601
  • Ticks - Using ticks is not recommended and is not well supported with LINQ.
  • ISO8601 - Default format for this provider.
  • JulianDay - JulianDay format, which is what SQLite uses internally
• JournalMode - Determines how SQLite handles the transaction journal file.  Under the covers this calls “PRAGMA journal_mode={0}”. Default “Delete”
  • Delete - default mode, this causes SQLite to create and destroy the journal file as-needed.
  • Persist - When this is set, SQLite will keep the journal file even after a transaction has completed.  It’s contents will be erased, and the journal re-used as often as needed.  If it is deleted, it will be recreated the next time it is needed.
  • Off - This option disables the rollback journal entirely.  Interrupted transactions or a program crash can cause database corruption in this mode!
• DefaultIsolationLevel - Sets the default isolation level for transactions on the connection. Default “Serializable”. Note this is a standard .Net “System.Data.IsolationLevel” type. Do not even try to change this . Only “Serializable” and “ReadCommitted” can be used, all others will throw a “NotSupportedException”. However for completeness the full list and meanings are outlined below:
  • Serializable - A range lock is placed on the DataSet, preventing other users from updating or inserting rows into the dataset until the transaction is complete.
  • ReadUncommitted – A dirty read is possible, meaning that no shared locks are issued and no exclusive locks are honored.
  • ReadCommitted - Shared locks are held while the data is being read to avoid dirty reads, but the data can be changed before the end of the transaction, resulting in non-repeatable reads or phantom data.
  • RepeatableRead - Locks are placed on all data that is used in a query, preventing other users from updating the data. Prevents non-repeatable reads but phantom rows are still possible.
  • Snapshot - Reduces blocking by storing a version of data that one application can read while another is modifying the same data. Indicates that from one transaction you cannot see changes made in other transactions, even if you requery.
  • Chaos - The pending changes from more highly isolated transactions cannot be overwritten.
  • Unspecified - A different isolation level than the one specified is being used, but the level cannot be determined.
• Version – Sets the default version of the SQLite engine to instantiate.  Currently the only valid value is 3, indicating version 3 of the SQLite library.
• Synchronous – Sets the synchronization mode (file flushing) of the connection string.  Default is “Normal”.  Under the covers this calls “PRAGMA synchronous={0}”. Supported values are:
  • Normal – Normal file flushing at critical sections of the code
  • Full – Full file flushing after every write operation
  • Off – Use the default operating system’s file flushing, SQLite does not explicitly flush the file buffers after writing
• UseUTF16Encoding – Boolean indicator setting the encoding for the connection.  The default is “False” which indicates UTF-8 encoding.
• Pooling – Sets whether or not to use connection pooling.  The default is “False”
• BinaryGUID - Sets to or not to store GUID’s in binary format.  The default is True which saves space in the database.
• Deprecated Uri – depreciated call to the data source property.
• DefaultTimeout – sets the default command timeout for newly-created commands.  This is especially useful for commands used internally such as inside a SQLiteTransaction, where setting the timeout is not possible. Time is in seconds, the default is 30 seconds.
• Enlist – Determines whether or not the connection will automatically participate in the current distributed transaction (if one exists). Default is True.
• LegacyFormat – If enabled, uses the legacy 3.xx format for maximum compatibility, but results in larger database sizes. Default is False. Under the covers this calls “PRAGMA legacy_file_format={0}”.

So this is the connection string options available on the latest release. The interesting part is how I found this – in the source code (so at least now you dont need to do that part). However the next blog will be how to build the provider to help you diagnose and understand any issues you come across.

Series Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

 SQLiteConnection.CreateFile("c:\\mydatabasefile.db3"); SQLiteConnection.CreateFile("c:\\mydatabasefile.db3");

So starting with the biggies:

1) Limited Database enforced Type Safety:

CREATE TABLE  IF NOT EXISTS Book
(
BookISBN     varchar(14) primary key
,Name         varchar(200)
,PublisherId    int
);

INSERT INTO BOOK(BookISBN,Name)
VALUES(
     "ReallyLongValueThatShouldCauseAProblem"
    , "CSharpHacker SQLite Manual");

Most DBs honor the restrictions of the character lengths. SQLite will record the information you submit to it – even it is technically breaks your DDL definitions. More about this can be read about it Datatypes In SQLite Version 3. What should specifically be called out is:

Any column in a version 3 database, except an INTEGER PRIMARY KEY column, may be used to store any type of value.

So a more outlandish example would be:

DROP TABLE IF EXISTS Issue;
CREATE TABLE  IF NOT EXISTS Issue
(
 IssueId 	int
,Description	varchar(200)
);
INSERT INTO Issue(IssueId,Description)
VALUES(
     "ReallyLongTextValueThatShouldCauseAProblem"
     , "CSharpHacker SQLite Manual");
SELECT * from Issue;

If you run the above you get “ReallyLongTextValueThatShouldCauseAProblem|CSharpHacker SQLite Manual”, with no errors!
Is that the sound of DBA’s running away screaming in horror…? Believe it or not this was intentional, and ‘sometimes’ sensible . I’ll cover a little more on that later, but lets go full circle on this.

Underneath the covers there is a 64 bit row identifier attached to every row, but if you want to use your own you can override that to use your own values. If you do this then this column categorically has to be an INTEGER value. So looking at the SQL below you can see we have now changed the type from ‘int’ to ‘INTEGER PRIMARY KEY’. This now enforces you have a value that is an integer for this one column, and the below fails with a “SQL error near line xx: datatype mismatch”.

DROP TABLE IF EXISTS Issue2;
CREATE TABLE  IF NOT EXISTS Issue2
(
 IssueId 	INTEGER Primary key
,Description	varchar(200)
);
INSERT INTO Issue2(IssueId,Description)
VALUES(
     "ReallyLongTextValueThatShouldCauseAProblem"
    , "CSharpHacker SQLite Manual");
SELECT * from Issue2;

So one final warning “INTEGER PRIMARY KEY” is the same as “integer primary key”, but not “int primary key”. If you are wanting to override the internal 64 bit row identifier you have to use “integer” and not “int”, “bigint”, “short integer”. Thats an easy one to miss! Final quirk is that the “INTEGER PRIMARY KEY” needs to be exactly that or “INTEGER PRIMARY KEY ASC” (case insensitive), but “INTEGER PRIMARY KEY DESC” will not use the internal ID.

2. Only one process can write to the DB at the same time

The locking in SQLite is not fine grained! If anyone is writing to the DB it locks everyone and everything else out of the DB. All readers and writers are blocked until the pending write is done. So if you are expecting a lot of concurrent writes to the database that may need to be interleaved with some reads from other threads/processes you may experience some blocking. In reality it shouldnt matter too much as the writes are normally speedy. It should also be noted that in SQLite v2 there was a possibility of writer starvation where so many reads are going on the writer was never able to lock the file to perform the write – this has been addressed in SQLite V3.

3. Limited support for ALTER TABLE

Now I bet this has driven off the DBA’s now! Because of its heritage SQLite team hasn’t ever needed to support the more complex data migrations supported by other database products. You can use to add a column to the end of a table or to change the name of a table. However if you want to make more complex changes in the structure of a table, you have to recreate a new table and then copy the contents of your old table into the new one.

4. Foreign Keys are ‘recorded’ but not enforced

Technically the same thing can be achieved using the ‘Triggers’, but you need to be aware that these are ‘soft RI’ keys and are not enforced by the database engine.

5. Support for UPSERT via “REPLACE” or “INSERT OR REPLACE”

For MySQL users they are familiar with REPLACE, SQLite has aliased REPLACE to its native “INSERT OR REPLACE’.

So if you are still reading that is good! This is definitely the scary page that drives a few DBA’s into fits, but its sensible to know what you are using and its behaviors before it surprises you.

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

However back to the question – How does it stack up?

As you can see SQL Server definitely has the most features, in fact as far as features go its a slam dunk. However often feature count just isn’t everything. The major drawback for SQL Server is that it requires a Server process to run, and that then means maintenance, probably a DBA and all of a sudden things have got complicated (and expensive)! So we come back to the question “What are we looking to do with the database?”. If the answer is heavy duty processing that requires stored procedures, cubes and large amount of concurrent remote access by users – SQL Server is definitely your answer. However I have to think that you are reading this because SQL Server is too heavy weight for your needs. So on to the other candidates.

SQL CE is a Microsoft database that started in the mobile space and moved up to the PC platform. Overall is has reasonable features but is known to be slower and heavier than SQLite. It definitely has some strengths in the SQL Server ==> SQL CE replication, but outside of that its largely even or less featured that SQLite. If you are looking for replication definitely checkout SQL CE, but I would also say the Microsoft Synch framework is evolving and we may see a native SQLite provider (or determine that the ADO.NET is sufficient) in this area.

SQLite is a lightweight add-on to C# (System.Data.SQLite.DLL ~ 850Kb). You may note this is heavier than the often referenced SQLite runtimes take “200Kb”, but in this 850Kb (457Kb Zipped) is the SQL DLL, ADO.NET SQLite provider and the magic glue to bring these two worlds together. If you are using .Net you can assume you have to assume your package size will go up by 457Kb, and the runtime size on disk will be ~850Kb. Performance of SQLite ranks it above SQL CE for insert tests, and it has a lower file size – and the database can be assessed on a massive number of operating systems!

Before you jump whole heartedly on the SQLite bandwagon, it is only fair to also point out SQLite has some quirks (SQLite folks call these ‘features’ – I’m not going to argue, but its approach is different in a couple of cases). I’m going to cover these in the next blog entry “Part 5″. These are things that you definitely need to be aware of rather than discover!

The key question that the article should be asking is “What type of application features benefits from SQLite?”. The best way I can describe SQLite is that is is an “Application Data Store” that you then have ANSI-92 SQL access to. This removes the needs of rolling your own indexing into files, creating convoluted file system ‘databases’ etc. If you can wrap your head around how you can use SQLite to your advantage I think you will be pleasantly surprised.

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

In addition to bug fixes, this release includes the following major changes from 2.4.0:

• We now ship Microsoft’s ASP.NET MVC (http://www.asp.net/mvc) stack and ASP.NET MVC applications can run with Mono.
• xbuild has been updated
• Removed non-free amqp0-8.xml documentation.

More detailed information can be found “http://www.mono-project.com/Release_Notes_Mono_2.4.2“. I have to say the MVC part is really exciting for me, again for cloud and cross OS capabilities this seems a winner!

Now on to the other side of the news – I almost blogged about this earlier, but it seems good prudence as we now can have the good news of the new release with the less good of people saying don’t use it!

Richard Stallman is recommending against using C# as a platform for free software in his “Why free software shouldn’t depend on Mono or C#“. This obviously got a big slashdot following at http://news.slashdot.org/story/09/06/27/1759255/Richard-Stallman-Says-No-To-Mono. While I understand his sentiments I dont believe that his thoughts on Microsoft pulling C# support using patents is any more of a threat than any other language (aka everything has patent exposure – its not language specific). Actually I’ve been very impressed with Microsoft recently on their move towards interoperability and their more open support of open systems. Adding fanning the flames was this Here we go again – why Mono doesn’t suck. Ah I had forgotten about the joys of how much can be written about a emotive topic in such a short period of time

Gareth

• Mono on the iPhone…. To say this is exciting is an understatement. Unfortunately I’m a little afraid that Apple wont like it too much (read allow it in their TOS).
  

  “3.3.2 An Application may not itself install or launch other executable code by any means, including without limitation through the use of a plug-in architecture, calling other frameworks, other APIs or otherwise. No interpreted code may be downloaded and used in an Application except for code that is interpreted and run by Apple’s Published APIs and built-in interpreter(s)”

  .Mono is normally an interpreted language very similar to Java and the TOS explictly disallows running byte code on the machine.  So this could be VERY interesting, or it could be a nice concept and not never to hit the Apple store. Definitely watch this space!Otherwise we will end up with a compiler than generates Object C code

• Firefox 3.5 should be out tomorrow.
• Another EDW appliance comes into the fold with Aster Data.
• Simple problems highlighted with “SQL Server 2005 reporting in management studio“.

Gareth