• WAL (Write Ahead Logging) support – this means:
  • Generally faster
  • More concurrency
  • Better disk performance when processing the WAL logs.
  • Cant use network shares or different computers for writing to the same logs.
  • [Complete WAL information can be found here]
• Query planner enhancement
• Logical database size is now stored in the database header so that bytes can be appended to the end of the database file

Well done to the SQLite team!

Gareth

PS And for those wondering, I will restart blogging again on a more regular basis very soon, just been unburying myself!

• [How to find out what procedure is doing a data modification]
  • Very simple, yet effective approach to determine what RPC call is altering your data. This is a read it once, you get the ah-ha why didn’t I think of that moment and you wont need to reference it again!
• [Temporary procedures : T-SQL]
  • We all know about temporary tables, but you can do that for stored procedures as well. Neat if you have a need for it
• [Conor vs. FOREIGN KEY join elimination]
  • Explains a bit of the history why a single column foreign key performs better than a multi-column FK. Its the old 80:20 rule at work, or probably more accurately 99.9:0.1 rule
• [SSRS -Report execution failed. Solution: SSPI=NTLM]
  • Seems downgrading to NTLM solves some integrated security issues if Kerberos is not playing nicely. Handy to know.
• [Take advantage of Table Variables NOT being transactional]
  • Ever needed to store state in a transaction that you didnt want to be rolled back? Think log tables! Read and learn
• [SQL SERVER – Fragmentation, Detection and how to resolve it]
  • Nice overview of fragmentation, and shows how to use DMV’s rather than the old school SQL 2000 DBCC approaches
• [CodePlex: SYS2 DMVs]
  • Hopefully this will serve as a good collection point – its very ‘young’ at the moment, but I would like it to live!
• [Locking improvements in SQL 2008 R2]
  • Handy to know about, unfortunately R2 has a price hike associated with it
• [Find the most expensive operations in Execution plans]
  • Handy way to take the XML plan and process it in a way to get a tabular list of the most expensive queries back. Very nice.

I suspect everyone who has dealt with scalar UDF functions in production environments are already VERY aware of the performance sucking capability they can have on your nice server. Here are some nice comments if you are not painfully aware:

• [Do Scalar UDFs give SQL Server a Bad Name?]
• [UDF Overhead – A simple example]

Gareth

• Cloud
• Authorization
• Death of smart clients
• No-SQL and ORM
• BI becomes commodity
• Netbooks – and the Google ‘netbook’
• Deployment
• SVN ==> GIT or Mercurial

Cloud

No surprises here, with Amazon running this model ‘forever’, Google not too far behind and now even Microsoft has started to chime in as well. The are alliances and definitions for SaaS, IaaS, PaaS. This concept is here to stay, and in fact will affect us in significant ways.

Authorization/Authentication

This is the big play that will help make the cloud be really viable, and even work with managing external companies that need to work for your company (think PCI compliance and support). This model will be based on the claims concept that has been knocking around for a while now, all it needed was something to push it into the limelight. Clouds have started to push authorization / authentication! In addition when this matures to a ‘tipping point’ I have no doubt customers will start to ‘outsource’ their claims trust (with contractual backup!) to other companies. For example if your company needs to provide PCI support to a customer, today the customer will have to be provided the peoples names and assign them logon accounts and manage their passwords. This becomes a major overhead and can be relatively easily managed through a claims based authorization system. Naturally this wont happen within two years – but it is coming!

Death of Smart Clients

Companies will realize that Smart Clients (aka Prism, Click-Once web forms) are only a stop gap. Browsers will rule the application space within 5 years, and will have made significant inroads before 2012. Conceptually they all make sense, but in today’s platform varied world they are dead – but it seems not everyone realizes it yet!

No-SQL / ORM

It seems that there are two factors pushing these forwards, but the root of it is that the historical relational database has so much overhead associated with infrastructure, management and DBA’s – people are looking for alternatives. It has become increasingly common for application programmers to fear the rules of relational databases they have to integrate with as they have evolved into their own discipline. This leads into ORM and which really try to ‘ignore’ the storage mechanism and provide plain business value. The No-SQL group are in a similar boat, they are interested in scaling and maintainance and know traditional relational models don’t meet their business needs.

BI becomes a Commodity

Microsoft has been playing the “BI is coming, BI is coming” song for a while now. However it seems with the SQL 2008 R2 release we are starting to get commodity solutions. This in intern will coerce the competitors to step up. The good news for competitors is that Microsoft has got more pricey, its no longer the ‘cheap kid on the block’. Open Source is starting to catch up with its commercial cousins. Either way its going to be an interesting couple of years for BI.

Netbooks – and the Google one

We have heard the term many times before, and even had a couple of false starts. The difference now is:

1. Processing power has significantly cheapened – allowing more inexpensive offerings
2. Web/Cloud infrastructure is in place. No worries about email or docs being stored on line.

So we have pretty good netbooks now, but I suspect Google will potentially set the standard for the netbooks. If done right (as opposed to the Nexus!) they have a killer combination. This is one I’m really interested to see what happens!

Deployment

Vista didnt get much traction and XP is on its last officially supported legs, as such I suspect that companies will be rolling out Windows 7 over the next 2-3 years. This will stress the deployment tools, and require individuals that know deployment to step in. In addition to this Windows Server 2008 R2 only comes in 64 bit – software houses realize the time has come to admit they are going to have to invest in supporting 64 bit solutions. This is going to be unusual for customers as they have forgotten how to roll out such changes as it has been a while since the last great deployment upgrade. There will be lots of opportunities in this area!

SVN ==> GIT / Mercurial

It seems that the CVS savior is in the process of being usurped by the DVCS products. Specially GIT and Mercurial are the strong runners. Its a shame because I had waited a while for SVN to mature to a point and now GIT & Mercurial are eating up projects. Oddly at the time of writing this even the Microsoft sponsored CodePlex site now supports Mercurial !

Finally the one I didn’t add to the list is software parallelism programming models. I suspect this will mature with all the new multi-core processors that are coming out. However I think this will mature after 2012

Gareth

Isn’t “Secure String” an oxymoron for .Net? So if we are thinking about securing some sensitive data in say C or C++ its relatively simple load it into a char array memory and encrypt it, wiping the memory out after the information has been loaded.

Now try that with .Net! From the Microsoft site:

“A String is called immutable because its value cannot be modified once it has been created.“

So how can you destroy one? Set it to empty? Well simply put you can’t . Once your string is not longer referenced, or worse yet your object containing the string its time for the Garbage Collector to come and do its work. The problem is if your object has been around long enough to get into Generation 1 or 2 then it is going to take a bit longer.

Hmmm so in translation if you keep a password, Credit Card, encryption key or some other sensitive text in memory as a string you cant destroy it (think memset for us oldies!). Only the GC can free the memory for you, and you are dependent on HOW it frees that memory. I personally don’t know for a fact if it memsets it to blank, or just dereferences the pointer. However I would be willing to bet it is the option that requires the least amount of work and that doesn’t bode well for controlling the exposure of our sensitive data.

Plainly that proverbially sucks!

Enter the “SecureString” class, from the MS site it says:

“Represents text that should be kept confidential. The text is encrypted for privacy when being used, and deleted from computer memory when no longer needed.”

Wow doesn’t that just sound like the ticket we need! Secure, Encryption, delete from memory – how fantastic! Uh oh keep reading the remarks:

“Your application can render the instance immutable and prevent further modification by invoking the MakeReadOnly method.

…

Use appropriate members of the System.Runtime.InteropServices..::.Marshal class, such as the SecureStringToBSTR method, to manipulate the value of a SecureString object.”

BSTR – oh I feel the COM headache coming back!

Actually its really not that bad, but its definitely not a straight swap for a System.String. See some example code below:

using System;
using System.Text;
using System.Runtime.InteropServices;
using System.Security.Cryptography;
using System.Security;

namespace CSharpHacker.Utilities
{
   /// <summary>
   /// Demo helper class for <see cref="SecureString"/>
   /// </summary>
   public class SensitiveDataHelper
   {
      private SecureString sensitiveData;

      /// <summary>
      /// Gets or sets the sensitive data class from helper
      /// </summary>
      /// <value>The sensitive data container.</value>
      public SecureString SensitiveData
      {
         get { return sensitiveData; }
         set { sensitiveData = value; }
      }

      /// <summary>
      /// UNSECURE: Converts the secured sensitive data into an unsecured string.
      /// </summary>
      /// <remarks>
      /// This is a dangerous function that converts secured information into
      /// unsecured data.
      /// </remarks>
      /// <returns>String representing the secured data</returns>
      public string SensitiveDataToString()
      {
         IntPtr ptr = Marshal.SecureStringToGlobalAllocUnicode(this.sensitiveData);
         try
         {
             // Unsecure managed string
             return Marshal.PtrToStringUni(ptr);
         }
         finally
         {
             Marshal.ZeroFreeGlobalAllocUnicode(ptr);
         }
      }

      /// <summary>
      /// UNSECURE: Base64s encodes a SHA512 has of the sensitive data
      /// </summary>
      /// <returns>SHA512 hash value</returns>
      public string Base64SensitiveDataHash()
      {
         IntPtr bstr = Marshal.SecureStringToBSTR(sensitiveData);
         try
         {
            // Pretend simple hash function that returns a string - this is fake just for readability!!
            string output = Marshal.PtrToStringBSTR(bstr);

            Marshal.FreeBSTR(bstr);

            SHA512 sha = new SHA512Managed();
            byte[] result = sha.ComputeHash(Encoding.UTF8.GetBytes(output));
            return Convert.ToBase64String(result);
         }
         finally
         {
            Marshal.ZeroFreeBSTR(bstr);
         }
      }

      /// <summary>
      /// Loads the sensitive data into a <see cref="SecureString"/>
      /// </summary>
      /// <param name="sensitiveInformation">The sensitive information to protect.</param>
      public void LoadSensitiveData(char[] sensitiveInformation)
      {
         try
         {
            using (SecureString securePassword = new SecureString())
            {
               foreach (char c in sensitiveInformation)
               {
                  securePassword.AppendChar(c);
               }
               securePassword.MakeReadOnly();
               this.sensitiveData = securePassword.Copy();
            }
         }
         finally
         {
            // discard the char array
            Array.Clear(sensitiveInformation, 0, sensitiveInformation.Length);
         }
      }

      /// <summary>
      /// Loads the sensitive data into a <see cref="SecureString"/>
      /// </summary>
      /// <param name="sensitiveInformation">The sensitive information to protect.</param>
      public void LoadSensitiveData(string sensitiveInformation)
      {
         char[] sensitive = new char[sensitiveInformation.Length];
         sensitiveInformation.CopyTo(0, sensitive, 0, sensitive.Length);

         LoadSensitiveData(sensitive);
      }

      /// <summary>
      /// Loads the sensitive data into a <see cref="SecureString"/>
      /// </summary>
      /// <param name="sensitiveInformation">The sensitive information to protect.</param>
      public void LoadSensitiveData(StringBuilder sensitiveInformation)
      {
         char[] sensitive = new char[sensitiveInformation.Length];
         sensitiveInformation.CopyTo(0, sensitive, 0, sensitive.Length);

         LoadSensitiveData(sensitive);
      }

      /// <summary>
      /// Creates the secure string from string.
      /// </summary>
      /// <param name="unprotectedSensitiveInformation">The unprotected sensitive information.</param>
      /// <returns></returns>
      public static SecureString CreateSecureStringFromString(string unprotectedSensitiveInformation)
      {
         char[] unprotectedSensitive = unprotectedSensitiveInformation.ToCharArray();
         SecureString secureInformation = new SecureString();
         try
         {
            foreach (char c in unprotectedSensitive)
            {
               secureInformation.AppendChar(c);
            }
            secureInformation.MakeReadOnly();
            return secureInformation;
         }
         finally
         {
            // discard the char array
            Array.Clear(unprotectedSensitive, 0, unprotectedSensitive.Length);
         }
      }
   }
}

A word of caution to the above code:

• SensitiveDataToString – is considered insecure as it returns a string of the encrypted data. The same data we are trying to encrypt! However it is a commonly requested function, and so there is the implementation.
• Base64SensitiveDataHash – is considered insecure as it currently uses a temporary string (:-( ), at this time I don’t have a way to converts a BSTR into a an array without going through a string first. One way would be to process it prior to being made ReadOnly, or alternatively someone can write a comment how to convert a C# BSTR into a byte array!

So even with all those disclaimers running a program that takes input will still a ‘leak information’ for the System.String. Specifically the tricky area is how do you get them into you program in the first place? Read them from a database, WinForm user input, or a web page? Kinda tricky ! If you search the web there are implementations of  secure login controls that build it up character by character, but certainly something to think about.

So how Secure is “SecureString”? The answer is “it depends”, but reasonably secure and a heck of a lot better than System.String. A while ago there was a big storm about tools that can connect to the process and decrypt your SecureStrings. The best rebuttal to this I’ve seen can be read about in [SecureString Redux]. I definitely recommend reading this.

Now I have to say I’ve been meaning to write this for some time now! Hopefully this helped raise awareness of the string leakage risks in the .Net language and ways to help minimize the string information leak scenario.

Potential enhancements to the helper class would be:

• Make the Hashing really secure, and allow a “HashAlgorithm” to be passed in
• Allow external encryption
• Allow secure serialization of data (via the encryption)

Gareth

• [OWASP Code Review]
• [How To: Perform a Security Code Review for Managed Code]
• [Phase 1: Conduct a Security Design Review]
• [Security Code Reviews]

Note as other are suggested, or I find others I’ll update the list.

So for all us who have spent time searching, or had pages pointed out to us , for options  this is an excellent new years present!

Many thanks to the SQLite guys for both their product, and their handy dandy search feature!

“Google Wave is an online tool for real-time communication and collaboration. A wave can be both a conversation and a document where people can discuss and work together using richly formatted text, photos, videos, maps, and more.”

This is the email that hit my in box across the weekend.

After playing with the software me its like a communal no holds barred whiteboard – but with history tracking. However this whiteboard is written text, graphics, videos etc. People can read or help contribute/refine the whiteboard – which is known in Google terms as a “Wave”. People can be invited late into the Wave, and quickly catchup (or back track depending on your perspective) using the history tracking feature. A friend described is as “cross between a wiki/twitter/blog/im/email.” Its certainly an interesting concept and I can actually see it working well in a number of cases.

Playing with it is certainly revealing and you can quickly see the high level implications of it, but you definitely need either a small group of folks needing to solve a “problem” (be it a BBQ cook out that needs organizing, or a technical design) to really benefit from this collaboration tool (at this time), or you need more adoption to get people to use it more.

Frankly this is pretty exciting – I just need to get a reason/project to use it in more anger now

More information can be found at this [About Google Wave], and the key to its success will be its “Add-in” extensions and the underlying API. I believe the most valuable parts of this whole thing is the API backbone used to power “Wave”, and how far browsers can be pushed to support these very dynamic collaborative applications.

The standard warning however is, this is not production, nor it is even “Beta”, this is “Preview”! So dont put all your eggs in the basket without a backup/backout plan!

Gareth

• [What's New in SQL Server 2008 R2 Editions]
  • Biggest first – Price increase! To say I’m disappointed is an understatement in this area. Specifically for developers  already have to support 2005 & 2008. Now they have to deal with R2, and persuade their customers to move to R2 if they want certain features.
  • MDS – Master Data Services – more on this one later. Pretty cool
  • Parallel Data Warehouse release
    • The DATAllegro purchase is now seeing the light as a Microsoft product!
    • Support of very large datasets (100Tb-1Pb). This is definitely something I would like to play with, but at $57,496 per processor…
  • Standard now has backup compression – but with the price tag…
• [SQL Server 2008 R2 November CTP – What’s New In Reporting Services?]
  • Report Builder 3.0 is available! Woohoo!
  • Enhanced sharepoint integration
  • Aggregates of aggregates
  • New Data Visualization Report Items
• [Using Filtered Statistics with Partitioned Tables]
  • Learn to appreciate filtered statistics!
• [MDS (Master Data Services) is x64 Only]
  • I totally get this, but this is the tipping point for me to repave my machines
  • With this requiring x64 and many new products requiring Windows 2008, Windows 7 or Vista it seems now is the time to move up.
• [MSSQL End of service dates for Service packs]
  • Support for SQL Server 2005 Service Pack 2 (SP2)  will end on January 12, 2010
  • Support for SQL Server 2008 RTM will end on April 13, 2010

More to come!

Gareth

[http://www.codersrevolution.com/index.cfm/2009/10/21/Sequoia-Voting-System-Witch-Hunt-err-Study-Project"]

The learning we can take away from this is if you don’t adequately cleanse then you can expect the data to become available! While its an interesting concept they apparently tried (and not too successfully) to do. The best way to clean a database is to create a new one and just copy in the data you want exposed. Don’t trust the handy dandy DROP/DELTE

If they wanted to expose/publish the 88 tables, then they should have created a new DB, copied in the tables and released it. Anything less than that you have to be VERY careful! And for the more security conscious it would be created on a recently wiped drive on a recently rebooted computer!

Once a fighters weakness has been exposed there is really nothing you can do to unhide that weakness. You could have the best fighter in the world one day, then the weakness is exposed… You are in trouble!

Security is very much the same. You can perform all the scans, probes, fuzzes, code reviews and feel confident (well as confident anyone does in the security world!) that you are pretty well covered. One revelation a day later can completely invalidate your expectations, and you have to completely start over. Sometimes it is a slow build up, other times it is the equivalent of a bomb.

Bottom line is once a weakness has been exposed you need to:

• See if it can be simply covered
  • Fighter can learn to defend against take downs (or not get hit in the head )
  • Algorithm can be enhanced to extend its life DES==>3DES
• Relegate
  • Fighter acts as the ‘gatekeeper’ to the higher competition levels
  • Algorithms security clearance has been lowered, it cant be used in the more secure areas. Examples of this are theoretical discoveries that are likely to result in the actual weakness discover some time later.
• Retire
  • Fighter retires, becomes a commentator!
  • Algorithm depreciated as it is shown to be fundamentally insecure, now studied in university to show the weakness that designers need to be aware of. Think WEP!

If the weakness is known it is natural the opponent will attempt to get a competitive advantage using it. The longer the weakness is known the more adept the opposition will be at exploiting it.  This is true for both MMA & security!

Companies running a SDL are the equivalent to the fighters stable. It is their job to recognize the weaknesses and manage the processes and algorithms so any weaknesses are covered or retired before they become a major problem.

Gareth

• [MSSQL - The Query Optimizer and Parameter Sniffing]
  • If you dont know about query sniffing, or came across it a couple of years ago and have forgotten. Give this a (re)read.
  • The key here is the “Optimize” for a typical parameter, I dont recall this existing in 2000. So if you are stuck with 2000 (you know who you are!), this obviously wont work!
• [Need to protect your C# code? Have a look at nCloak]
  • Article covering how nCloak does naming.
  • This isnt production ready, but if you have spare time it would be interesting to see how far this project can go.
  • This shows the benefit that Mono is bringing to the C# world!
• [Just got onto TFS? Ready to try GIT/Mercurial? Read more about branch strategies in DVCS]
  • Seems like only yesterday everyone was moving from VSS to TFS. Now GIT and Mercurial are on the scenes.
  • This article covers various strategies for CI or even “Promiscuous Integration”
  • Interestingly the DVCS (Distributed Version Control Systems) appear better suited to OSS projects than internal corporate ones.
• [Microsoft SDL Developer Starter Kit]
  • The Microsoft SDL Developer Starter Kit provides a compliation of baseline developer security training materials on core Microsoft Security Development Lifecycle (SDL) topics.
• [OSSEC - open source Host-based Intrusion Detection System (HIDS)]
  • Its free! 2.2 came out September 8, 2009
  • It has a powerful correlation and analysis engine, integrating log analysis, file integrity checking, Windows registry monitoring, centralized policy enforcement, rootkit detection, real-time alerting and active response.
• [Microsoft releases mini-Fuzzer & Binary analyzer]
  • Finally , MS have released some simple fuzzers to help developers understand what they are facing from the black hats!
• [IT executive going to China? If you follow the guidance it will be expensive!]
  • Paraphrasing this short article wont do it justice.  Among the measures it recommends to IT executives regarding the protection of their computer equipment when traveling to that country are (wow is about all I can say!):
  • • Leave your standard IT equipment at home – buy separate gear to use in China
    • Weigh the machine before you go and when you get back
    • “Clean” thoroughly the equipment (re-image the laptop you used)
    • Throw away the mobile phone you used during your stay.
• [A Shortage of Technical Managers]
  • This just made me smile!

So looking at what SQLite3.exe has too offer it pretty much supports it out of the box. Very nice

Requirements:

• Loading speed
• Making the data to consuming applications available asap

While I love C# and frankly its hard to go back to C or C++, sometimes performance trumps the creature comforts we have become accustomed to.

Note: I did this without circling back to a C# implementation as I know the data and performance requirements  are tight and in this case I wanted max performance with no code! The biggest factor to a successful implementation is to ensure you use the tools best for the job, not just the ones you favor in that specific year.

So first things first – create a table to take the input

DROP TABLE IF EXISTS BookSales;
CREATE TABLE IF NOT EXISTS BookSales
(
   Store    int
  ,Date     varchar
  ,OrderReference varchar
  ,Line     int
  ,BookISBN varchar(14)
  ,Quantity int
  ,Price    int
, Primary Key (OrderReference,Line)
);

Next is the magic. We need to load the CSV into the table:

.separator "|"
.import BookSales.txt BookSales

Wow that was easy . You can see we set the separator to be a pipe rather than comma in this case, then the import.

.IMPORT [FileName] [Table]

Now the database is ready to be queried! But if we want to take it just one stage further:

.output SummaryBookSales.csv
SELECT Store, Date, BookISBN, SUM(Quantity), SUM(Price)
FROM BookSales
GROUP BY Store, Date, BookISBN;

Now we output the results of our simple aggregation into a pipe separated output file.

Tying this all together in a single configuration file, which we will call “BookAnalysisLoader.sql”, gives us:

DROP TABLE IF EXISTS BookSales;
CREATE TABLE IF NOT EXISTS BookSales
(
   Store    int
  ,Date     varchar
  ,OrderReference varchar
  ,Line     int
  ,BookISBN varchar(14)
  ,Quantity int
  ,Price    int
, Primary Key (OrderReference,Line)
);

.separator "|"
.import BookSales.txt BookSales

.output SummaryBookSales.csv
SELECT Store, Date, BookISBN, SUM(Quantity), SUM(Price)
FROM BookSales
GROUP BY Store, Date, BookISBN;
.exit

The last piece of the puzzle is the final execution:

sqlite3.exe BookSalesAnalysis.db3 < BookAnalysisLoader.sql

Now we have a newly created database with our analysis data in it, and we have a summary CSV file generated from the output. So we can load the CSV into Excel or another DB, or directly interrogate the DB for more analytical information – and all without coding!

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

Well firstly I have to say its not that easy , secondarily it really shows the true benefit of [TDD] – more on that later. So on to the discoveries!

Being a TDD advocate I started off with an interface and then the tests before the code. So far so good – then on to the code. As a Spongebob episode would say “Several hours later” we were done. So crank up the tests and we are looking golden.

Unfortunately the code coverage wasn’t half as good as I had hoped for. So next couple of hours was spent updating the tests for coverage, and a number of edge cases surrounding the throwing of exceptions for invalid parameters. Interestingly a couple of areas that showed up outside of the TDD approach were certain areas in “for loops” where not executed because my test case was really unknowingly best case. The part that was revealing to me was that the TDD is really a “business” driven approach, code coverage is believed to be a programmers indicator of  ‘quality’. So while TDD (at least the first pass I end up doing) meets the business needs it is very hard to get 100% coverage from a business scenario.

Finally I got it up to 99.84% coverage. This last standout was VERY annoying, specifically looking at the code coverage everything was green – no red or yellow sections. Hrrrmm. Ok it was a simple function that has this 1 block of unexecuted code – a switch statement taking a enum in as a parameter.  So lets have a look have a look (note I’ve stripped a lot of the content and comments out for readability:

   public enum PasswordStrengthIndex
   {
      None = 0,
      Weak = 1,
      Medium = 2,
      Strong = 3,
      MostStrong = 4,
   }

      public void ResetToDefinedPolicyStrength(PasswordStrengthIndex strength)
      {
         switch (strength)
         {
            case PasswordStrengthIndex.None:
               MinimumPasswordLength = 0;
               break;
            case PasswordStrengthIndex.Weak:
               MinimumPasswordLength = 4;
               break;
            case PasswordStrengthIndex.Medium:
               MinimumPasswordLength = 6;
               break;
            case PasswordStrengthIndex.Strong:
               MinimumPasswordLength = 8;
               break;
            case PasswordStrengthIndex.MostStrong:
               MinimumPasswordLength = 12;
               break;
         }
         return;
      }

The test case iterated through the enums and verified the output matched the expectations. Still I had this annoying block that was unexecuted. The next step was the Jimmy Neutron “Think Think Think”, a-ha! No default handler – so add one. Hmmm I cant add one that actually can be exercised as I’ve already used all my values in my enum. Well adding a default handler to anyone of those case statements still didnt resolve the issue. Now it was getting personal! I was dangerously close to calling in technical gurus and venting my disgust! So before making the calls time to search Google for “Code Coverage Switch” – and boy Google never ceases to amaze me.

Firstly it confirmed my suspicion that is was indeed the switch statement, second was a link that explained it in detail. Basically the default handler was not getting called – and there were a number of ‘breaking’ workarounds (change the enum to have another element to be used in the default case) in the Google answers, but one one that I ‘liked’ the most was to force the cast (which I would link out to the original author out of respect but I cant find it now ).

      public void ResetToDefinedPolicyStrength(PasswordStrengthIndex strength)
      {
         switch (strength)
         {
            case PasswordStrengthIndex.None:
               MinimumPasswordLength = 0;
               break;
            case PasswordStrengthIndex.Weak:
               MinimumPasswordLength = 4;
               break;
            case PasswordStrengthIndex.Medium:
               MinimumPasswordLength = 6;
               break;
            case PasswordStrengthIndex.Strong:
               MinimumPasswordLength = 8;
               break;
            case PasswordStrengthIndex.MostStrong:
               MinimumPasswordLength = 12;
               break;
            default:
               throw new ArgumentException("Supplied strength is not recognized as enum", "strength");
         }
         return;
      }

      [TestMethod()]
      [ExpectedException(typeof(ArgumentException))]
      public void ResetToDefinedPolicyStrengthBogus()
      {
         PasswordPolicy policy = new PasswordPolicy();
         policy.ResetToDefinedPolicyStrength(<strong>(PasswordStrengthIndex)666</strong>);
      }

Look at the last line of the listing above. Use a cast to force the enum to be a number out side of the range it was looking for – ta da!  On a partially related note this reminds me of the fact Enums are definitely interesting beasts that can bite you if you have compiled against one then change its definition in a different assembly.

So now we have got to 100% code coverage – woo hoo! The realization (which is obvious and well know to many) is that without TDD you would most likely end up writing tests that cover your code – but miss the business requirements. So any programmers out there who think they have good code coverage but are not using TDD are likely to have a false believe in their code because the tool says 100% green (unless they are coding guru’s!) – they are only really testing what they have written and not the business requirements. On the flipside just writing TDD tests without coverage is highly unlikely to meet all the ‘code’ edge cases (different than ‘business’ edge cases). So the conclusion it that the blend of the two are the best, but yet still not perfect. You have to start with TDD, and then move to code coverage – business first, code second.

I’m hoping at some point management teams will start to understand that even with 100% executed code coverage there can still be bugs as its a data world we live in! Obviously I’m an optimist!

Gareth

For starters I recommend reading the article [http://en.wikipedia.org/wiki/Password_strength]. This is a good article covering the relative strengths of passwords, and gives a guide for determining the strength of a random password and a human derived password.

The major problem with passwords are that humans need to remember them, or they write them down. In an interesting technology twist historically you only used to have to worry about your co-workers having access/abusing your password because there was implicit physical security in place – you could only log on if you were physically in the office.  As such at that time your biggest threat was your co-workers, unfortunately the secondary defense of physical location has effectively been removed with the internet and VPN technology.  So now your threat count has increased from the people you work with to the entire world! Add to this these people are financially motivated and can directly target you – its a whole lot scarier out there now!

So before jumping into the implementations we need to go through well known things to avoid to help improve password strength:

• Avoid sequences – keyboard or alphabet based (abcd, qwert, 1234, !@#$% etc)
• Avoid dictionary words, especially common ones! Be aware that common misspellings are also used in dictionary based attacks – so unless your misspelling is VERY unusual then you can expect it to be in a dictionary!
• Avoid leet/1337 password substitution of words (eg P@ssw0rd, M1cr0$0ft, 0\/\/n3d). Again these are now all in dictionaries, so while it may be harder to brute force – they are pretty trivial for a dictionary attack. Of course it doesnt hurt to be 1337, but it just really doesn’t help defend a targeted attack.
• Avoid team names, socials, license names etc.

Things to avoid to minimize compromise exposure:

• Use different passwords for different online accounts
• Avoid using information about you that can be readily be found on the web as a password reset scheme. DOB, where you were born, school name etc.
• If any account needs the most rigorous password control it is your email account. Nearly every online system ties back to an email account. If you need to reset a password, it normally goes to your email address. If that is compromised then that is really the opening of Pandora’s box.

Alright, lets start with the weakest ‘safe’ approach – Information entropy:

• This strength calculation only holds true for ‘random’ passwords. No human (at least that I know) can really generate a random password on their own. The best approach that I’m aware of is to start up notepad and get your two year old to start smacking your keyboard. Then take this text and randomly change case of characters and inserting special characters. Unfortunately this is still weak because we have 2 hands and the keyboard is naturally divided into where your hands go. This generation is not as randomly distributed as people would think – nor would I recommend it! But at least you have a starting point, but then you have to write it down!
• [0-9] – 10 possible symbols per character – 3.32 bits of base2 log entropy
• [a-z] – 26 possible symbols per character- 4.7 bits of base2 log entropy
• [A-Z] – 26 possible symbols per character- 4.7 bits of base2 log entropy
• [A-Z, 0-9] – 36 possible symbols per character- 5.17 bits of base2 log entropy
• [A-Z,a-z] – 52 possible symbols per character- 5.7 bits of base2 log entropy
• [A-Z, a-z, 0-9] – 62 possible symbols per character- 5.95 bits of base2 log entropy
• [A-Z, a-z, 0-9, Special] – 94 possible symbols per character – 6.55 bits of base2 log entropy

So we can see that having a strong password using completely random information will be hard to generate on our own, yet this approach is what is what is most commonly used to in web applications to determine password strength. This is not strong enough because humans are naturally not random. Using this theory the following non-random passwords generate results that imply the passwords are strong:

• 12345678901234567890 – 20*3.32 => 66.4 bits of entropy
• !!!!!!!!!!!!!!!!!!!! – 10 * 6.55 => 65.5 bits of entropy
• !@#$%^&*() – 10 & 6.55 => 65.5 bits of entropy
• qwertyuiop[]qwertyuiop[] = 24 * 6.55 => 157.2 bits of entropy.

The more astute among us will see the last two passwords were generated by running your finger across the a keyboard line of on a US keyboard. To enter the 24 characters password took under 3 seconds. So if anyone saw someone entering a password like this at work or in a library – its pretty easy to duplicate. Plainly you can see that with human users they are going to opt for the easiest way to remember and enter a password – this will never be random!

So to help avoid our users from becoming victims we have to try to take away the ‘easy’ passage from them. We have to assume the password is not going to be mathematically random – so we need to start from a different position. We have to ensure we remove the human weaknesses that other ‘black hats’ are looking to exploit.

So going back to the beginning of the article we are going to create an interface to define a ‘password policy’ that provides us a way to help enforce a stronger passwords – or  at least allows systems to setup a common language for handling passwords.

   /// <summary>
   /// Interface for defining a password policy
   /// </summary>
   /// <remarks>
   /// This security policy determines whether passwords
   /// meet pre-determined complexity requirements.
   ///
   /// If this policy is enabled, passwords must meet the
   /// following minimum requirements:
   ///
   /// Not contain the user's account name or parts of the
   /// user's full name that exceed four consecutive
   /// characters.
   /// Be at least <see cref="MinimumPasswordLength"/>
   /// characters in length
   /// Contain characters from three of the following
   /// four categories:
   /// English uppercase characters (A through Z)
   /// English lowercase characters (a through z)
   /// Base 10 digits (0 through 9)
   /// Non-alphabetic characters (for example, !, $, #, %)
   ///
   /// Complexity requirements are enforced when passwords
   /// are changed or created.
   /// </remarks>
   public interface IPasswordPolicy : IPolicy
   {
      /// <summary>
      /// Indicates the minimum password strength index for
      /// this policy (see PasswordStrengthIndex)
      /// </summary>
      /// <remarks>
      /// This value is based of a calculation of
      /// information entropy after sequences
      /// and dictionary words have been
      /// removed.
      /// </remarks>
      /// <value>
      /// The minimum index of the password strength.
      /// </value>
      PasswordStrengthIndex MinimumPasswordStrengthIndex
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the minimum length of the password.
      /// </summary>
      /// <value>The minimum length of the password.</value>
      int MinimumPasswordLength
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the maximum length of the password.
      /// </summary>
      /// <value>The maximum length of the password.</value>
      int MaximumPasswordLength
      {
         get;
         set;
      }

      /// <summary>
      /// If policy requires mixed case
      /// </summary>
      /// <value>true if policy needs mixed case</value>
      bool RequireMixedCase
      {
         get;
         set;
      }

      /// <summary>
      /// If policy needs digits
      /// </summary>
      /// <value>true if policy needs digits.</value>
      bool RequireDigits
      {
         get;
         set;
      }

      /// <summary>
      /// If policy needs special characters
      /// </summary>
      /// <value>
      /// true if require special characters are needed
      /// </value>
      bool RequireSpecialCharacters
      {
         get;
         set;
      }

      /// <summary>
      /// Indicates if the username needs to be additionally
      /// supplied to verify the password complexity against
      /// </summary>
      /// <value>
      /// true require username to check password against
      /// </value>
      bool RequireUsernameToCheckPasswordAgainst
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the maximum count of characters
      /// in a sequence
      /// </summary>
      /// <value>The maximum count of characters
      /// in a sequence.</value>
      int MaximumCharacterSequenceCount
      {
         get;
         set;
      }

      /// <summary>
      /// The duration of the lockout in minutes.
      /// </summary>
      /// <remarks>
      /// This security setting determines the number of
      /// minutes a locked-out account remains locked
      /// out before automatically becoming unlocked.
      /// The available range is from 0 minutes through
      /// 99,999 minutes.
      /// If you set the account lockout duration to less
      /// than zero, the account will be locked out until an
      /// administrator explicitly unlocks it. If an account
      /// lockout threshold is defined, the account lockout
      /// duration must be greater than or equal to
      /// the reset time.
      /// </remarks>
      /// <value>The duration of the lockout.</value>
      int LockoutDuration
      {
         get;
         set;
      }

      /// <summary>
      /// Gets or sets the lockout threshold.
      /// </summary>
      /// <remarks>
      /// This security setting determines the number of
      /// failed logon attempts that causes a user
      /// account to be locked out. A locked-out
      /// account cannot be used until it is reset
      /// by an administrator or until the
      /// lockout duration for the account has expired. You
      /// can set a value between 0 and 999 failed
      /// logon attempts. If you set the value to 0,
      /// the account will never be locked out.
      /// </remarks>
      /// <value>The lockout threshold.</value>
      int LockoutThreshold
      {
         get;
         set;
      }

      /// <summary>
      /// Reset account lockout after X minutes
      /// </summary>
      /// <remarks>
      /// This security setting determines the number of
      /// minutes that must elapse after a failed logon
      /// attempt before the failed logon attempt
      /// counter is reset to 0 bad logon attempts.
      /// The available range is 1 minute to
      /// 99,999 minutes.
      /// </remarks>
      /// <value>The duration of the lockout.</value>
      int LockoutResetInMinutes
      {
         get;
         set;
      }
   }

You can see this password policy template extends the initial outline to not only provide guidance for the number of entropy bits, but allows for the policy to cover the lock out strategy in the case of incorrect password handling and password expiry approaches. If you look at the source code you will also see the options that are available, but for the sake of this article we are trying to keep on point

So on to the actual strength testing, this oddly is rather simple at the end of the day. We are going to use an Interface definition (IPassword) for the Password processor (makes testing & mocking easier) so we can actually have multiple implementations (think MEF!).  Now the actual implementation.

1. Check for sequences using various lookup tables to determine if any sequences exist. If a sequence length is detected, and is longer than allowed the password fails the policy. The tables include:
   • Alphabetic + numeric sequence
   • QWERTY US Keyboard
   • QWERTY UK Keyboard
   • AZERTY Keyboard
2. Perform simple DecodeEliteEncoding then perform a simple hardcoded dictionary match of well known super common passwords
3. If supplied (and if required) compare password elements to the user name

The end implementation is still fairly simple and it would be fairly easy to improve on this implementation.  The most obvious ones are to support a custom dictionary and add more custom keyboard sequences. Other extensions would be to store the passwords and become a real password token service. We can leave it up to the reader to provide an implementation of IPassword to call the Google password rating service rather than the above implementation:

https://www.google.com/accounts/RatePassword?Passwd=csharphacker

All good stuff! I hope this helps (and the source code) people provide a better approach to helping strengthen passwords.

[Download source code here]

The linked source code is liable to change over time so check back often. The source code uses the Microsoft testing framework and currently has 100% code coverage! Although I don’t think 100% is all that people think it is.

Finally the goal is to make everything a more secure place – and in reality the best approach is to use a strong memorable password in conjunction with a hardware token that changes every minute.

As always feedback is welcome!

Gareth

• Improved query planner:
  • Through better use of statistics
  • Compile time option enables Analyze to better handle the index histograms
  • Additionally it was just plain improved as well!
• Recursive triggers
• Delete triggers fire during a REPLACE/MERGE
• More precise use of caching approaches with the
  • Shared Cache – basically cache the results within the application rather can on each thread. Now configurable on a per-thread basis rather than global.
  • Private – The old way

Well done to the SQLite team! Good stuff

More details can be found [SQLite Release 3.6.18 On 2009 Sep 11 (3.6.18)]

• VMware announces VMware vCloud Express, goes head to head with Amazon EC2
  • VMware today announced vCloud Express, a new class of service that will deliver on-demand, pay-as-you-go computing power as a service, much like Amazon Web Services’ Elastic Compute Cloud (EC2).
• A PCI-Compliant Cloud? Not at Amazon
  • Very interesting debate and conclusion regarding Amazon EC2 & PCI. Worth a read!
• Azure Reference Architecture Explored with Project Riviera
  • If you’ve been wanting to see how a multi-tenant architecture works with Windows Azure Platform, you’ll want to see Project Riviera. The project has been released on MSDN and includes source code. Interestingly the sample application is a loyalty application!
• Cloud Cartography & Side Channel Attacks
  • This is obviously dependent on the availability of a weakness in the hypervisor, but definitely an interesting concept none the less.

Security

• Microsoft Internet Information Server (IIS) FTP Server Buffer Overflow Lets Remote Authenticated Users Execute Arbitrary Code
  • Any one running IIS 5, 5.1, or 6 and FTP – watch out!
• New version of WiKID authentication server
  • WiKID Systems announced version 3.4 of the WiKID Strong Authentication Server in Enterprise and Community Editions. New features include built-in support for the SAML Single Sign-On Service for Google Apps, a new self-registration process for Active Directory users and extended vendor-specific RADIUS attributes.
  • Pretty interesting stuff, not seen this before.

Hopefully my writers block has been solved and I’ll get back to writing some code

Gareth

Definitely pretty cool stuff. Now to see how well it performs and verify it supports all the nice SQ L stuff  (sparse columns, PIVOT, stored procedures etc).

This will certainly be hyped by Microsoft over the coming weeks, and we will be trying it out! Much more to come on this topic,

Gareth

• [The Virtualization Nation Podcast – Episode 3: Want to boot a physical computer from a VHD?]
• [Step-By-Step: Turning a Windows 7 DVD or ISO into a Bootable VHD Virtual Machine]

Seriously cool – I mean seriously! Native booting to a VHD on your disk – I want to buy who ever thought of that and got management to agree to putting that into Windows 7 (think HyperVisor) a beer!

I LOVE IT!!

Firstly it should be said that the standard GZipStream stream provides the functionality I’m sure the MS engineers expected it to do, which was for HTTP based compression (at least I think that was its expected purpose). However it is certainly not a fully featured class that is really easy to use for the programmers looking to get quick & helpful access to the GZip compression.

Specifically the problem I needed to solved was I needed to know how big any given “.GZ” decompressed file was without fully reading and decompressing the file. It seemed trivial enough – “gzip.exe -l” does what I needed, but no amount of hunting within MSDN helped. So on to the ever handy GZip wikipedia entry that detailed enough of the file format and provided the reference to the “GZIP file format specification version 4.3“.

So armed this this information we can start to decode the GZip file format to extract the length. Infact this class will check the file to see if it is GZip compressed and returns the decompressed length for that or the regular file length if it is not compressed.

The following class functions have been implemented (see the bottom of the article for the link to the full project):

   /// <summary>
   /// Utility class to help with managing GZip (.gz) files in .Net
   /// </summary>
   /// <remarks>
   /// This is a trivial wrapper class on top of <see cref="GZipStream"/> that does a little magic
   /// under the covers by looking at the underlying data format and retrieves the
   /// stored data information within the GZip compressed file.
   /// </remarks>
   public class GZipHelper
   {
      /// <summary>
      /// Gets the compressed file details
      /// </summary>
      /// <param name="filename">The filename.</param>
      /// <returns>True if file exists, else false</returns>
      public bool GetFileDetails(string filename);

      /// <summary>
      /// Gets the compressed file information from a file stream
      /// </summary>
      /// <param name="fileStream">The file stream.</param>
      /// <remarks>
      /// Definitions provided by RFC 1952 -GZIP File Format Specification (May 1996).
      /// Coding was performed against ftp://ftp.isi.edu/in-notes/rfc1952.txt
      /// </remarks>
      public void GetFileInformation(FileStream fileStream);

      /// <summary>
      /// Compresses the file file
      /// </summary>
      /// <param name="filename">The filename.</param>
      /// <param name="overWriteExisting">if set to <c>true</c> [over write existing].</param>
      /// <returns></returns>
      public void CompressFile(string filename, bool overWriteExisting);

      /// <summary>
      /// Decompresses the file.
      /// </summary>
      /// <param name="filename">The filename.</param>
      /// <param name="overWriteExisting">if set to <c>true</c> [over write existing].</param>
      /// <returns></returns>
      public bool DecompressFile(string filename, bool overWriteExisting);

      /// <summary>
      /// Returns a seekable stream into either a file or compressed file (defaults read-only)
      /// </summary>
      /// <remarks>
      /// Decompresses the stream into a <see cref="MemoryStream"/> if the file is compressed
      /// otherwise just returns back a regular <see cref="FileStream"/> as a <see cref="Stream"/>
      /// </remarks>
      /// <param name="filename">The filename to open.</param>
      /// <returns>Reference to opened stream</returns>
      public Stream GetSeekableStream(string filename);
   }

In combination to this the following properties are available:

• CompressedLength – Size of the compressed file (or regular file size if not compressed)
• DecompressedLength – Size of the file if it were uncompressed (or regular file size if not compressed)
• IsTextFile – Indicates if GZip thought the file was text based, potentially leading to better compression
• CompressionModeValue – Numeric indication of the compression mode used
• CRC16Present – Indicates a CRC16 is available for the file
• ExtraFieldsPresent – Additional meta fields are available in the file
• FileNamePresent – GZip contains the original file name
• FileCommentPresent – Compressed file has a comment associated with it
• IsCompressed – Indicates if the file is GZip compressed or not
• CompressedDate – If stored this is the date the file was compressed.
• CRC32 – CRC32 value associated with the file

Along with the project there are MSTest harnesses to test the class (trivial implementations). So the features of the class are:

• Can trivially determine a true file size (regardless if it was compressed via GZip or is uncompressed). This makes your code path much more readable if you are dealing with mixed file types.
• Provides a Seekable stream into the compressed file via via a MemoryStream. The key is that you dont need to worry about the compression (unless you are reading in BIG files) as you will get back a Stream for either a File or a Compressed file – both support seeking. This can be handy if you problem assumes it can Seek in the stream and you need to access GZip files!
• Trivial Decompress file, this also honors the CompressedDate. If that date is set then the decompressed file has that creation date.
• Trivial Compress file. Unfortunately at the time of writing I’ve not updated the header to include the date of the compressed file. This may come in a later version (and if so I’ll update the blog – but definitely no promises!).

Simple example usages are (taken straight from the unit tests!):

// Perform a file compression
GZipHelper actual = new GZipHelper();
actual.CompressFile(_fileName, true);

// Perform a file decompression
GZipHelper actual = new GZipHelper();
string fileName = "CSharpHackerSmallTest.txt.gz";
actual.DecompressFile(fileName, true);

// Get a seekable stream
GZipHelper actual = new GZipHelper();
using (Stream dataStream = actual.GetSeekableStream("CSharpHackerSmallTest.txt.gz"))
{
    // Silly seek - but it just shows it can be done
    dataStream.Seek(0, SeekOrigin.Begin);
    StreamReader sr = new StreamReader(dataStream);
    string contents = sr.ReadToEnd();

    Assert.AreEqual(119, contents.Length);
}

// Gets natural decompressed file length from a compressed file.
GZipHelper actual = new GZipHelper();
actual.GetFileInformation("CSharpHackerSmallTest.txt.gz");
Assert.AreEqual(119, actual.DecompressedLength);

Finally it should be noted that by all accounts the standard implementation of GZipStream in the base .Net libraries (actually the DeflateStream) has a problem when attempting to compress random or already compressed data. There is a Microsoft Connect article [http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=93930] that details the issue.

The GZipStream and DeflateStream classes can _significantly_ increase the size of “compressed” data. That means, they don’t just add a few header bytes as stand-alone compressors do, but they _inflate_ the data by as much as 50%. This is apparently because these classes do not check for incompressible data which is a standard feature of all stand-alone compressors. Both classes work fine when the data actually can be compressed.

Please refer to this thread for more details:

http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=179704&SiteID=1

The base implementation worked for me and met my specific needs without the need of bringing in any third party DLLs. Which incidentally also has a nice benefit for those looking to bring this into proprietary software of avoiding any licensing discussions with supervisors! If you want a more robust GZipStream implementation you can check out http://dotnetzip.codeplex.com/. This apparently has a drop in replacement, but this class could still be useful even if use this drop in replacement as well.

I hope this helps some one out there

[Download GZipHelper (Source + Project) Here]

This download link will always have the latest and greatest version.

Gareth