Isn’t “Secure String” an oxymoron for .Net? So if we are thinking about securing some sensitive data in say C or C++ its relatively simple load it into a char array memory and encrypt it, wiping the memory out after the information has been loaded.

Now try that with .Net! From the Microsoft site:

“A String is called immutable because its value cannot be modified once it has been created.“

So how can you destroy one? Set it to empty? Well simply put you can’t . Once your string is not longer referenced, or worse yet your object containing the string its time for the Garbage Collector to come and do its work. The problem is if your object has been around long enough to get into Generation 1 or 2 then it is going to take a bit longer.

Hmmm so in translation if you keep a password, Credit Card, encryption key or some other sensitive text in memory as a string you cant destroy it (think memset for us oldies!). Only the GC can free the memory for you, and you are dependent on HOW it frees that memory. I personally don’t know for a fact if it memsets it to blank, or just dereferences the pointer. However I would be willing to bet it is the option that requires the least amount of work and that doesn’t bode well for controlling the exposure of our sensitive data.

Plainly that proverbially sucks!

Enter the “SecureString” class, from the MS site it says:

“Represents text that should be kept confidential. The text is encrypted for privacy when being used, and deleted from computer memory when no longer needed.”

Wow doesn’t that just sound like the ticket we need! Secure, Encryption, delete from memory – how fantastic! Uh oh keep reading the remarks:

“Your application can render the instance immutable and prevent further modification by invoking the MakeReadOnly method.

…

Use appropriate members of the System.Runtime.InteropServices..::.Marshal class, such as the SecureStringToBSTR method, to manipulate the value of a SecureString object.”

BSTR – oh I feel the COM headache coming back!

Actually its really not that bad, but its definitely not a straight swap for a System.String. See some example code below:

using System;
using System.Text;
using System.Runtime.InteropServices;
using System.Security.Cryptography;
using System.Security;

namespace CSharpHacker.Utilities
{
   /// <summary>
   /// Demo helper class for <see cref="SecureString"/>
   /// </summary>
   public class SensitiveDataHelper
   {
      private SecureString sensitiveData;

      /// <summary>
      /// Gets or sets the sensitive data class from helper
      /// </summary>
      /// <value>The sensitive data container.</value>
      public SecureString SensitiveData
      {
         get { return sensitiveData; }
         set { sensitiveData = value; }
      }

      /// <summary>
      /// UNSECURE: Converts the secured sensitive data into an unsecured string.
      /// </summary>
      /// <remarks>
      /// This is a dangerous function that converts secured information into
      /// unsecured data.
      /// </remarks>
      /// <returns>String representing the secured data</returns>
      public string SensitiveDataToString()
      {
         IntPtr ptr = Marshal.SecureStringToGlobalAllocUnicode(this.sensitiveData);
         try
         {
             // Unsecure managed string
             return Marshal.PtrToStringUni(ptr);
         }
         finally
         {
             Marshal.ZeroFreeGlobalAllocUnicode(ptr);
         }
      }

      /// <summary>
      /// UNSECURE: Base64s encodes a SHA512 has of the sensitive data
      /// </summary>
      /// <returns>SHA512 hash value</returns>
      public string Base64SensitiveDataHash()
      {
         IntPtr bstr = Marshal.SecureStringToBSTR(sensitiveData);
         try
         {
            // Pretend simple hash function that returns a string - this is fake just for readability!!
            string output = Marshal.PtrToStringBSTR(bstr);

            Marshal.FreeBSTR(bstr);

            SHA512 sha = new SHA512Managed();
            byte[] result = sha.ComputeHash(Encoding.UTF8.GetBytes(output));
            return Convert.ToBase64String(result);
         }
         finally
         {
            Marshal.ZeroFreeBSTR(bstr);
         }
      }

      /// <summary>
      /// Loads the sensitive data into a <see cref="SecureString"/>
      /// </summary>
      /// <param name="sensitiveInformation">The sensitive information to protect.</param>
      public void LoadSensitiveData(char[] sensitiveInformation)
      {
         try
         {
            using (SecureString securePassword = new SecureString())
            {
               foreach (char c in sensitiveInformation)
               {
                  securePassword.AppendChar(c);
               }
               securePassword.MakeReadOnly();
               this.sensitiveData = securePassword.Copy();
            }
         }
         finally
         {
            // discard the char array
            Array.Clear(sensitiveInformation, 0, sensitiveInformation.Length);
         }
      }

      /// <summary>
      /// Loads the sensitive data into a <see cref="SecureString"/>
      /// </summary>
      /// <param name="sensitiveInformation">The sensitive information to protect.</param>
      public void LoadSensitiveData(string sensitiveInformation)
      {
         char[] sensitive = new char[sensitiveInformation.Length];
         sensitiveInformation.CopyTo(0, sensitive, 0, sensitive.Length);

         LoadSensitiveData(sensitive);
      }

      /// <summary>
      /// Loads the sensitive data into a <see cref="SecureString"/>
      /// </summary>
      /// <param name="sensitiveInformation">The sensitive information to protect.</param>
      public void LoadSensitiveData(StringBuilder sensitiveInformation)
      {
         char[] sensitive = new char[sensitiveInformation.Length];
         sensitiveInformation.CopyTo(0, sensitive, 0, sensitive.Length);

         LoadSensitiveData(sensitive);
      }

      /// <summary>
      /// Creates the secure string from string.
      /// </summary>
      /// <param name="unprotectedSensitiveInformation">The unprotected sensitive information.</param>
      /// <returns></returns>
      public static SecureString CreateSecureStringFromString(string unprotectedSensitiveInformation)
      {
         char[] unprotectedSensitive = unprotectedSensitiveInformation.ToCharArray();
         SecureString secureInformation = new SecureString();
         try
         {
            foreach (char c in unprotectedSensitive)
            {
               secureInformation.AppendChar(c);
            }
            secureInformation.MakeReadOnly();
            return secureInformation;
         }
         finally
         {
            // discard the char array
            Array.Clear(unprotectedSensitive, 0, unprotectedSensitive.Length);
         }
      }
   }
}

A word of caution to the above code:

• SensitiveDataToString – is considered insecure as it returns a string of the encrypted data. The same data we are trying to encrypt! However it is a commonly requested function, and so there is the implementation.
• Base64SensitiveDataHash – is considered insecure as it currently uses a temporary string (:-( ), at this time I don’t have a way to converts a BSTR into a an array without going through a string first. One way would be to process it prior to being made ReadOnly, or alternatively someone can write a comment how to convert a C# BSTR into a byte array!

So even with all those disclaimers running a program that takes input will still a ‘leak information’ for the System.String. Specifically the tricky area is how do you get them into you program in the first place? Read them from a database, WinForm user input, or a web page? Kinda tricky ! If you search the web there are implementations of  secure login controls that build it up character by character, but certainly something to think about.

So how Secure is “SecureString”? The answer is “it depends”, but reasonably secure and a heck of a lot better than System.String. A while ago there was a big storm about tools that can connect to the process and decrypt your SecureStrings. The best rebuttal to this I’ve seen can be read about in [SecureString Redux]. I definitely recommend reading this.

Now I have to say I’ve been meaning to write this for some time now! Hopefully this helped raise awareness of the string leakage risks in the .Net language and ways to help minimize the string information leak scenario.

Potential enhancements to the helper class would be:

• Make the Hashing really secure, and allow a “HashAlgorithm” to be passed in
• Allow external encryption
• Allow secure serialization of data (via the encryption)

Gareth

Once a fighters weakness has been exposed there is really nothing you can do to unhide that weakness. You could have the best fighter in the world one day, then the weakness is exposed… You are in trouble!

Security is very much the same. You can perform all the scans, probes, fuzzes, code reviews and feel confident (well as confident anyone does in the security world!) that you are pretty well covered. One revelation a day later can completely invalidate your expectations, and you have to completely start over. Sometimes it is a slow build up, other times it is the equivalent of a bomb.

Bottom line is once a weakness has been exposed you need to:

• See if it can be simply covered
  • Fighter can learn to defend against take downs (or not get hit in the head )
  • Algorithm can be enhanced to extend its life DES==>3DES
• Relegate
  • Fighter acts as the ‘gatekeeper’ to the higher competition levels
  • Algorithms security clearance has been lowered, it cant be used in the more secure areas. Examples of this are theoretical discoveries that are likely to result in the actual weakness discover some time later.
• Retire
  • Fighter retires, becomes a commentator!
  • Algorithm depreciated as it is shown to be fundamentally insecure, now studied in university to show the weakness that designers need to be aware of. Think WEP!

If the weakness is known it is natural the opponent will attempt to get a competitive advantage using it. The longer the weakness is known the more adept the opposition will be at exploiting it.  This is true for both MMA & security!

Companies running a SDL are the equivalent to the fighters stable. It is their job to recognize the weaknesses and manage the processes and algorithms so any weaknesses are covered or retired before they become a major problem.

Gareth

So looking at what SQLite3.exe has too offer it pretty much supports it out of the box. Very nice

Requirements:

• Loading speed
• Making the data to consuming applications available asap

While I love C# and frankly its hard to go back to C or C++, sometimes performance trumps the creature comforts we have become accustomed to.

Note: I did this without circling back to a C# implementation as I know the data and performance requirements  are tight and in this case I wanted max performance with no code! The biggest factor to a successful implementation is to ensure you use the tools best for the job, not just the ones you favor in that specific year.

So first things first – create a table to take the input

DROP TABLE IF EXISTS BookSales;
CREATE TABLE IF NOT EXISTS BookSales
(
   Store    int
  ,Date     varchar
  ,OrderReference varchar
  ,Line     int
  ,BookISBN varchar(14)
  ,Quantity int
  ,Price    int
, Primary Key (OrderReference,Line)
);

Next is the magic. We need to load the CSV into the table:

.separator "|"
.import BookSales.txt BookSales

Wow that was easy . You can see we set the separator to be a pipe rather than comma in this case, then the import.

.IMPORT [FileName] [Table]

Now the database is ready to be queried! But if we want to take it just one stage further:

.output SummaryBookSales.csv
SELECT Store, Date, BookISBN, SUM(Quantity), SUM(Price)
FROM BookSales
GROUP BY Store, Date, BookISBN;

Now we output the results of our simple aggregation into a pipe separated output file.

Tying this all together in a single configuration file, which we will call “BookAnalysisLoader.sql”, gives us:

DROP TABLE IF EXISTS BookSales;
CREATE TABLE IF NOT EXISTS BookSales
(
   Store    int
  ,Date     varchar
  ,OrderReference varchar
  ,Line     int
  ,BookISBN varchar(14)
  ,Quantity int
  ,Price    int
, Primary Key (OrderReference,Line)
);

.separator "|"
.import BookSales.txt BookSales

.output SummaryBookSales.csv
SELECT Store, Date, BookISBN, SUM(Quantity), SUM(Price)
FROM BookSales
GROUP BY Store, Date, BookISBN;
.exit

The last piece of the puzzle is the final execution:

sqlite3.exe BookSalesAnalysis.db3 < BookAnalysisLoader.sql

Now we have a newly created database with our analysis data in it, and we have a summary CSV file generated from the output. So we can load the CSV into Excel or another DB, or directly interrogate the DB for more analytical information – and all without coding!

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

For starters I recommend reading the article [http://en.wikipedia.org/wiki/Password_strength]. This is a good article covering the relative strengths of passwords, and gives a guide for determining the strength of a random password and a human derived password.

The major problem with passwords are that humans need to remember them, or they write them down. In an interesting technology twist historically you only used to have to worry about your co-workers having access/abusing your password because there was implicit physical security in place – you could only log on if you were physically in the office.  As such at that time your biggest threat was your co-workers, unfortunately the secondary defense of physical location has effectively been removed with the internet and VPN technology.  So now your threat count has increased from the people you work with to the entire world! Add to this these people are financially motivated and can directly target you – its a whole lot scarier out there now!

So before jumping into the implementations we need to go through well known things to avoid to help improve password strength:

• Avoid sequences – keyboard or alphabet based (abcd, qwert, 1234, !@#$% etc)
• Avoid dictionary words, especially common ones! Be aware that common misspellings are also used in dictionary based attacks – so unless your misspelling is VERY unusual then you can expect it to be in a dictionary!
• Avoid leet/1337 password substitution of words (eg P@ssw0rd, M1cr0$0ft, 0\/\/n3d). Again these are now all in dictionaries, so while it may be harder to brute force – they are pretty trivial for a dictionary attack. Of course it doesnt hurt to be 1337, but it just really doesn’t help defend a targeted attack.
• Avoid team names, socials, license names etc.

Things to avoid to minimize compromise exposure:

• Use different passwords for different online accounts
• Avoid using information about you that can be readily be found on the web as a password reset scheme. DOB, where you were born, school name etc.
• If any account needs the most rigorous password control it is your email account. Nearly every online system ties back to an email account. If you need to reset a password, it normally goes to your email address. If that is compromised then that is really the opening of Pandora’s box.

Alright, lets start with the weakest ‘safe’ approach – Information entropy:

• This strength calculation only holds true for ‘random’ passwords. No human (at least that I know) can really generate a random password on their own. The best approach that I’m aware of is to start up notepad and get your two year old to start smacking your keyboard. Then take this text and randomly change case of characters and inserting special characters. Unfortunately this is still weak because we have 2 hands and the keyboard is naturally divided into where your hands go. This generation is not as randomly distributed as people would think – nor would I recommend it! But at least you have a starting point, but then you have to write it down!
• [0-9] – 10 possible symbols per character – 3.32 bits of base2 log entropy
• [a-z] – 26 possible symbols per character- 4.7 bits of base2 log entropy
• [A-Z] – 26 possible symbols per character- 4.7 bits of base2 log entropy
• [A-Z, 0-9] – 36 possible symbols per character- 5.17 bits of base2 log entropy
• [A-Z,a-z] – 52 possible symbols per character- 5.7 bits of base2 log entropy
• [A-Z, a-z, 0-9] – 62 possible symbols per character- 5.95 bits of base2 log entropy
• [A-Z, a-z, 0-9, Special] – 94 possible symbols per character – 6.55 bits of base2 log entropy

So we can see that having a strong password using completely random information will be hard to generate on our own, yet this approach is what is what is most commonly used to in web applications to determine password strength. This is not strong enough because humans are naturally not random. Using this theory the following non-random passwords generate results that imply the passwords are strong:

• 12345678901234567890 – 20*3.32 => 66.4 bits of entropy
• !!!!!!!!!!!!!!!!!!!! – 10 * 6.55 => 65.5 bits of entropy
• !@#$%^&*() – 10 & 6.55 => 65.5 bits of entropy
• qwertyuiop[]qwertyuiop[] = 24 * 6.55 => 157.2 bits of entropy.

The more astute among us will see the last two passwords were generated by running your finger across the a keyboard line of on a US keyboard. To enter the 24 characters password took under 3 seconds. So if anyone saw someone entering a password like this at work or in a library – its pretty easy to duplicate. Plainly you can see that with human users they are going to opt for the easiest way to remember and enter a password – this will never be random!

So to help avoid our users from becoming victims we have to try to take away the ‘easy’ passage from them. We have to assume the password is not going to be mathematically random – so we need to start from a different position. We have to ensure we remove the human weaknesses that other ‘black hats’ are looking to exploit.

So going back to the beginning of the article we are going to create an interface to define a ‘password policy’ that provides us a way to help enforce a stronger passwords – or  at least allows systems to setup a common language for handling passwords.

   /// <summary>
   /// Interface for defining a password policy
   /// </summary>
   /// <remarks>
   /// This security policy determines whether passwords
   /// meet pre-determined complexity requirements.
   ///
   /// If this policy is enabled, passwords must meet the
   /// following minimum requirements:
   ///
   /// Not contain the user's account name or parts of the
   /// user's full name that exceed four consecutive
   /// characters.
   /// Be at least <see cref="MinimumPasswordLength"/>
   /// characters in length
   /// Contain characters from three of the following
   /// four categories:
   /// English uppercase characters (A through Z)
   /// English lowercase characters (a through z)
   /// Base 10 digits (0 through 9)
   /// Non-alphabetic characters (for example, !, $, #, %)
   ///
   /// Complexity requirements are enforced when passwords
   /// are changed or created.
   /// </remarks>
   public interface IPasswordPolicy : IPolicy
   {
      /// <summary>
      /// Indicates the minimum password strength index for
      /// this policy (see PasswordStrengthIndex)
      /// </summary>
      /// <remarks>
      /// This value is based of a calculation of
      /// information entropy after sequences
      /// and dictionary words have been
      /// removed.
      /// </remarks>
      /// <value>
      /// The minimum index of the password strength.
      /// </value>
      PasswordStrengthIndex MinimumPasswordStrengthIndex
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the minimum length of the password.
      /// </summary>
      /// <value>The minimum length of the password.</value>
      int MinimumPasswordLength
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the maximum length of the password.
      /// </summary>
      /// <value>The maximum length of the password.</value>
      int MaximumPasswordLength
      {
         get;
         set;
      }

      /// <summary>
      /// If policy requires mixed case
      /// </summary>
      /// <value>true if policy needs mixed case</value>
      bool RequireMixedCase
      {
         get;
         set;
      }

      /// <summary>
      /// If policy needs digits
      /// </summary>
      /// <value>true if policy needs digits.</value>
      bool RequireDigits
      {
         get;
         set;
      }

      /// <summary>
      /// If policy needs special characters
      /// </summary>
      /// <value>
      /// true if require special characters are needed
      /// </value>
      bool RequireSpecialCharacters
      {
         get;
         set;
      }

      /// <summary>
      /// Indicates if the username needs to be additionally
      /// supplied to verify the password complexity against
      /// </summary>
      /// <value>
      /// true require username to check password against
      /// </value>
      bool RequireUsernameToCheckPasswordAgainst
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the maximum count of characters
      /// in a sequence
      /// </summary>
      /// <value>The maximum count of characters
      /// in a sequence.</value>
      int MaximumCharacterSequenceCount
      {
         get;
         set;
      }

      /// <summary>
      /// The duration of the lockout in minutes.
      /// </summary>
      /// <remarks>
      /// This security setting determines the number of
      /// minutes a locked-out account remains locked
      /// out before automatically becoming unlocked.
      /// The available range is from 0 minutes through
      /// 99,999 minutes.
      /// If you set the account lockout duration to less
      /// than zero, the account will be locked out until an
      /// administrator explicitly unlocks it. If an account
      /// lockout threshold is defined, the account lockout
      /// duration must be greater than or equal to
      /// the reset time.
      /// </remarks>
      /// <value>The duration of the lockout.</value>
      int LockoutDuration
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the lockout threshold.
      /// </summary>
      /// <remarks>
      /// This security setting determines the number of
      /// failed logon attempts that causes a user
      /// account to be locked out. A locked-out
      /// account cannot be used until it is reset
      /// by an administrator or until the
      /// lockout duration for the account has expired. You
      /// can set a value between 0 and 999 failed
      /// logon attempts. If you set the value to 0,
      /// the account will never be locked out.
      /// </remarks>
      /// <value>The lockout threshold.</value>
      int LockoutThreshold
      {
         get;
         set;
      }

      /// <summary>
      /// Reset account lockout after X minutes
      /// </summary>
      /// <remarks>
      /// This security setting determines the number of
      /// minutes that must elapse after a failed logon
      /// attempt before the failed logon attempt
      /// counter is reset to 0 bad logon attempts.
      /// The available range is 1 minute to
      /// 99,999 minutes.
      /// </remarks>
      /// <value>The duration of the lockout.</value>
      int LockoutResetInMinutes
      {
         get;
         set;
      }
   }

You can see this password policy template extends the initial outline to not only provide guidance for the number of entropy bits, but allows for the policy to cover the lock out strategy in the case of incorrect password handling and password expiry approaches. If you look at the source code you will also see the options that are available, but for the sake of this article we are trying to keep on point

So on to the actual strength testing, this oddly is rather simple at the end of the day. We are going to use an Interface definition (IPassword) for the Password processor (makes testing & mocking easier) so we can actually have multiple implementations (think MEF!).  Now the actual implementation.

1. Check for sequences using various lookup tables to determine if any sequences exist. If a sequence length is detected, and is longer than allowed the password fails the policy. The tables include:
   • Alphabetic + numeric sequence
   • QWERTY US Keyboard
   • QWERTY UK Keyboard
   • AZERTY Keyboard
2. Perform simple DecodeEliteEncoding then perform a simple hardcoded dictionary match of well known super common passwords
3. If supplied (and if required) compare password elements to the user name

The end implementation is still fairly simple and it would be fairly easy to improve on this implementation.  The most obvious ones are to support a custom dictionary and add more custom keyboard sequences. Other extensions would be to store the passwords and become a real password token service. We can leave it up to the reader to provide an implementation of IPassword to call the Google password rating service rather than the above implementation:

https://www.google.com/accounts/RatePassword?Passwd=csharphacker

All good stuff! I hope this helps (and the source code) people provide a better approach to helping strengthen passwords.

[Download source code here]

The linked source code is liable to change over time so check back often. The source code uses the Microsoft testing framework and currently has 100% code coverage! Although I don’t think 100% is all that people think it is.

Finally the goal is to make everything a more secure place – and in reality the best approach is to use a strong memorable password in conjunction with a hardware token that changes every minute.

As always feedback is welcome!

Gareth

Firstly it should be said that the standard GZipStream stream provides the functionality I’m sure the MS engineers expected it to do, which was for HTTP based compression (at least I think that was its expected purpose). However it is certainly not a fully featured class that is really easy to use for the programmers looking to get quick & helpful access to the GZip compression.

Specifically the problem I needed to solved was I needed to know how big any given “.GZ” decompressed file was without fully reading and decompressing the file. It seemed trivial enough – “gzip.exe -l” does what I needed, but no amount of hunting within MSDN helped. So on to the ever handy GZip wikipedia entry that detailed enough of the file format and provided the reference to the “GZIP file format specification version 4.3“.

So armed this this information we can start to decode the GZip file format to extract the length. Infact this class will check the file to see if it is GZip compressed and returns the decompressed length for that or the regular file length if it is not compressed.

The following class functions have been implemented (see the bottom of the article for the link to the full project):

   /// <summary>
   /// Utility class to help with managing GZip (.gz) files in .Net
   /// </summary>
   /// <remarks>
   /// This is a trivial wrapper class on top of <see cref="GZipStream"/> that does a little magic
   /// under the covers by looking at the underlying data format and retrieves the
   /// stored data information within the GZip compressed file.
   /// </remarks>
   public class GZipHelper
   {
      /// <summary>
      /// Gets the compressed file details
      /// </summary>
      /// <param name="filename">The filename.</param>
      /// <returns>True if file exists, else false</returns>
      public bool GetFileDetails(string filename);

      /// <summary>
      /// Gets the compressed file information from a file stream
      /// </summary>
      /// <param name="fileStream">The file stream.</param>
      /// <remarks>
      /// Definitions provided by RFC 1952 -GZIP File Format Specification (May 1996).
      /// Coding was performed against ftp://ftp.isi.edu/in-notes/rfc1952.txt
      /// </remarks>
      public void GetFileInformation(FileStream fileStream);

      /// <summary>
      /// Compresses the file file
      /// </summary>
      /// <param name="filename">The filename.</param>
      /// <param name="overWriteExisting">if set to <c>true</c> [over write existing].</param>
      /// <returns></returns>
      public void CompressFile(string filename, bool overWriteExisting);

      /// <summary>
      /// Decompresses the file.
      /// </summary>
      /// <param name="filename">The filename.</param>
      /// <param name="overWriteExisting">if set to <c>true</c> [over write existing].</param>
      /// <returns></returns>
      public bool DecompressFile(string filename, bool overWriteExisting);

      /// <summary>
      /// Returns a seekable stream into either a file or compressed file (defaults read-only)
      /// </summary>
      /// <remarks>
      /// Decompresses the stream into a <see cref="MemoryStream"/> if the file is compressed
      /// otherwise just returns back a regular <see cref="FileStream"/> as a <see cref="Stream"/>
      /// </remarks>
      /// <param name="filename">The filename to open.</param>
      /// <returns>Reference to opened stream</returns>
      public Stream GetSeekableStream(string filename);
   }

In combination to this the following properties are available:

• CompressedLength – Size of the compressed file (or regular file size if not compressed)
• DecompressedLength – Size of the file if it were uncompressed (or regular file size if not compressed)
• IsTextFile – Indicates if GZip thought the file was text based, potentially leading to better compression
• CompressionModeValue – Numeric indication of the compression mode used
• CRC16Present – Indicates a CRC16 is available for the file
• ExtraFieldsPresent – Additional meta fields are available in the file
• FileNamePresent – GZip contains the original file name
• FileCommentPresent – Compressed file has a comment associated with it
• IsCompressed – Indicates if the file is GZip compressed or not
• CompressedDate – If stored this is the date the file was compressed.
• CRC32 – CRC32 value associated with the file

Along with the project there are MSTest harnesses to test the class (trivial implementations). So the features of the class are:

• Can trivially determine a true file size (regardless if it was compressed via GZip or is uncompressed). This makes your code path much more readable if you are dealing with mixed file types.
• Provides a Seekable stream into the compressed file via via a MemoryStream. The key is that you dont need to worry about the compression (unless you are reading in BIG files) as you will get back a Stream for either a File or a Compressed file – both support seeking. This can be handy if you problem assumes it can Seek in the stream and you need to access GZip files!
• Trivial Decompress file, this also honors the CompressedDate. If that date is set then the decompressed file has that creation date.
• Trivial Compress file. Unfortunately at the time of writing I’ve not updated the header to include the date of the compressed file. This may come in a later version (and if so I’ll update the blog – but definitely no promises!).

Simple example usages are (taken straight from the unit tests!):

// Perform a file compression
GZipHelper actual = new GZipHelper();
actual.CompressFile(_fileName, true);

// Perform a file decompression
GZipHelper actual = new GZipHelper();
string fileName = "CSharpHackerSmallTest.txt.gz";
actual.DecompressFile(fileName, true);

// Get a seekable stream
GZipHelper actual = new GZipHelper();
using (Stream dataStream = actual.GetSeekableStream("CSharpHackerSmallTest.txt.gz"))
{
    // Silly seek - but it just shows it can be done
    dataStream.Seek(0, SeekOrigin.Begin);
    StreamReader sr = new StreamReader(dataStream);
    string contents = sr.ReadToEnd();

    Assert.AreEqual(119, contents.Length);
}

// Gets natural decompressed file length from a compressed file.
GZipHelper actual = new GZipHelper();
actual.GetFileInformation("CSharpHackerSmallTest.txt.gz");
Assert.AreEqual(119, actual.DecompressedLength);

Finally it should be noted that by all accounts the standard implementation of GZipStream in the base .Net libraries (actually the DeflateStream) has a problem when attempting to compress random or already compressed data. There is a Microsoft Connect article [http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=93930] that details the issue.

The GZipStream and DeflateStream classes can _significantly_ increase the size of “compressed” data. That means, they don’t just add a few header bytes as stand-alone compressors do, but they _inflate_ the data by as much as 50%. This is apparently because these classes do not check for incompressible data which is a standard feature of all stand-alone compressors. Both classes work fine when the data actually can be compressed.

Please refer to this thread for more details:

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=179704&SiteID=1

The base implementation worked for me and met my specific needs without the need of bringing in any third party DLLs. Which incidentally also has a nice benefit for those looking to bring this into proprietary software of avoiding any licensing discussions with supervisors! If you want a more robust GZipStream implementation you can check out http://dotnetzip.codeplex.com/. This apparently has a drop in replacement, but this class could still be useful even if use this drop in replacement as well.

I hope this helps some one out there

[Download GZipHelper (Source + Project) Here]

This download link will always have the latest and greatest version.

Gareth

• MD5 - Dont use if you can avoid it as this is known to have vulnerabilities and should never be used!
• RIPEMD160 – This is supported by .Net, but isnt really heavily used. Recommend using SHA256 or SHA512
• SHA1- If you can avoid it use SHA256 or SHA512
• SHA2 Family
  • SHA256
  • SHA384
  • SHA512

If you haven’t come across MD5Sum.exe, SHA1Sum.exe, SHA256Sum.exe you can find native Windows ports here (or if you are looking for the more official GNU versions they can be found here) . Which if you are just looking for the command line tools that is probably enough. However sometimes you may have the need to do all this work yourselves in C#, if so this is the article that should help guide you!

First of all this is going to be fairly simple as the .Net library supports all of the above hash formats, so all we are really talking about doing is showing you the best way to use the supplied .Net runtimes to perform your hashing. So on to the magic (note to avoid width formatting issues this isnt exactly how I normally format the code!):

/// <summary>
/// Performs the SHA1 Hash function on file
/// </summary>
/// <param name="filename">
/// The filename to be hashed.
/// </param>
/// <returns>
/// SHA1 Hash value associated with the file
/// </returns>
public static string SHA1HashFile(string filename)
{
   string hashedValue = string.Empty;

   //create our SHA1 provider
   SHA1CryptoServiceProvider hashAlgorithm = new SHA1CryptoServiceProvider();

   //hash the data from the file
   byte[] hashedData = hashAlgorithm.ComputeHash(File.ReadAllBytes(filename));

   //loop through each byte in the returned byte array to convert into printed ASCII
   foreach (byte b in hashedData)
   {
      hashedValue += String.Format("{0,2:x2}", b);
   }

   //return the hashed value to the caller
   return hashedValue;
}

This does the SHA1 hashing of the supplied file – and matches the output of GNU version of SHA1Sum.exe. I told you it was simple . Dissecting this code should be pretty trivial:

1. Create the SHA1 Service provider (System.Security.Cryptography.SHA1CryptoServiceProvider)
2. Call ComputeHash passing in a byte array.
3. Take the results and output it as text. In case it wasnt obvious the results of the hash is a binary blob, hence the need to format it into a string friendly representation.

However there are really 2 problems with this code:

1. File.ReadAllBytes – This returns a byte array of the file – pretty much as you would expect! The problem is that if this is a very big file, for example a hash for a DVD, the entire file needs to be loaded into memory before it gets hashed. Obviously not the most optimal approach!
2. This is completely locked into SHA1, you need a new function for any other different hashing function. Not a biggie, but it would be nice to  get some reuse in now and then. Definitely useful if you see the full example where we have to use some fall back processing if an algorithm is not available.

Thankfully fixing this is still pretty trivial. To fix issue 1 rather than using the ComputeHash that takes in the byte array, use the one that takes the stream. This avoids the need for having the entire file in memory before the hash process can start. Out of curiosity I looked up the publicly available source code for the function to check it was in fact doing what I thought. Thankfully it is simple and obvious:

...
// Default the buffer size to 4K.
byte[] buffer = new byte[4096];
int bytesRead;
do {
   bytesRead = inputStream.Read(buffer, 0, 4096);
   if (bytesRead > 0) {
     HashCore(buffer, 0, bytesRead);
  }
} while (bytesRead > 0);
...

So we can see when we use the stream version of HashAlgorithm.ComputeHash Method (Stream) it only will use up a small memory chunk for calculating the hash values. So we are safe from big files from potentially killing the application.

Issue 2 – The .Net team did a nice job of creating base classes, one of which is HashAlgorithm. This is actually the class that implements the hashing ‘interface’. All hash algorithms must derive from this class. So we can use this to our advantage:

/// <summary>
/// Performs a Hash operation on the supplied file.
/// </summary>
/// <param name="filename">
/// The filename to be hashed.
/// </param>
/// <returns>
/// Selected Hash value associated with the file
/// </returns>
public static string HashFile(
          string filename
          , HashAlgorithm hashAlgorithm)
{
   if (!File.Exists(filename))
   {
      throw new ArgumentException(filename + " must exist", "filename");
   }

   string hashedValue = string.Empty;
   byte[] hashedData = null;

   // Create the stream
   using (FileStream fs = File.Open(filename, FileMode.Open, FileAccess.Read))
   {
      hashedData = hashAlgorithm.ComputeHash(fs);
   }

   //loop through each byte in the returned byte array
   foreach (byte b in hashedData)
   {
      //convert each byte and append
      hashedValue += String.Format("{0,2:x2}", b);
   }

   //return the hashed value
   return hashedValue;
}

Now we have a generic function to return back a hash value from any supported .Net hash algorithm. To call it you could just do “HashFile(filename,new SHA1CryptoServiceProvider())”. Voila! Performance and can trivially support any hashing class the .Net framework implements.

Ok so lets get a little more adventurous now. Attached to this entry is a very simple (aka not fully featured) HashSum source code that allows the same executable to be used to provide all the above hashing! However as a word of caution you need to be a little careful with the more advanced hashing. For example Microsoft supply both “SHA256CryptoServiceProvider” and “SHA256Managed”. On the surface they look pretty much the same, apart from SHA256CryptoServiceProvider is only available in .Net 3.5 (or higher). However since this uses Operating system cryptographic service providers they may not be available on the platform your program is running on. If that is the case then you will get the PlatformNotSupportedException exception thrown:

System.PlatformNotSupportedException was unhandled
  Message="The specified cryptographic algorithm is not supported on this platform."
  Source="System.Core"
  StackTrace:
       at System.Security.Cryptography.CapiNative.AcquireCsp(String keyContainer, String providerName, ProviderType providerType, CryptAcquireContextFlags flags, Boolean throwPlatformException)
       at System.Security.Cryptography.CapiHashAlgorithm..ctor(String provider, ProviderType providerType, AlgorithmId algorithm)
       at System.Security.Cryptography.SHA256CryptoServiceProvider..ctor()
       at CSharpHacker.Hash.HashSum.Main(String[] args) in X:\GIT\FileHash\FileSum.cs:line 76
       at System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)
       at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
       at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
       at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()
  InnerException:

Hmmm – you don’t read about that little chestnut in the MSDN help! So if you know you are going to only be running this on a Windows 2008, Vista or Windows 7 or later you can just use the “SHA256CryptoServiceProvider” version. However if you may have to support systems such as XP (potentially Windows 2003 as well) you will have to use the Managed versions. The safest route would be to provide a graceful fall back mechanism (possibly with a warning) that the CSP version could not be used and using the managed code version instead. This provides the best of both worlds, if the platform supports the CSP version you can use that (which should give you a speed increase) or you use the managed solution.

case "SHA256SUM":
default:
   try
   {
      hashAlgorithm = new SHA256CryptoServiceProvider();
   }
   catch (PlatformNotSupportedException)
   {
      // Fall back to the managed version if the CSP
      // is not supported on this platform.
      hashAlgorithm = new SHA256Managed();
   }
   break;

This is a second benefit of using the “HashAlgorithm” approach, the underlying code responsible for the generic hashing function is that it doesnt need to know what version (or even algorithm) it is using.

You also have to bear in mind that if you are going to use “SHA256CryptoServiceProvider” (or equivalent) you have to be using .Net 3.5 or greater.

Click to download Sample CSharp FileSum project

This is a .Net 3.5 project that based off the executable file name it uses that algorithm. So if you rename FileSum.exe to “SHA512Sum.exe” it will perform the SHA512 hash on the input file, MD5Sum.exe MD5 hash, etc. If the name is not a recognized name it defaults to using the SHA256 algorithm. This is not designed to be a wholesale replacement for SHA256Sum etc, but more of a guide how to write a fully featured version. So things missing from this include (and are not limited to ):

• No support for wildcards (simple enough to add but not there)
• No support for checking files match the input file (‘-c or –check’)
• Only binary mode is supported, no support for ASCII/Text mode. No support for ‘-t’ or ‘–text’
• No support for standard input processing

Hope you found this useful,

Gareth

is not supported