• WAL (Write Ahead Logging) support – this means:
  • Generally faster
  • More concurrency
  • Better disk performance when processing the WAL logs.
  • Cant use network shares or different computers for writing to the same logs.
  • [Complete WAL information can be found here]
• Query planner enhancement
• Logical database size is now stored in the database header so that bytes can be appended to the end of the database file

Well done to the SQLite team!

Gareth

PS And for those wondering, I will restart blogging again on a more regular basis very soon, just been unburying myself!

• [How to find out what procedure is doing a data modification]
  • Very simple, yet effective approach to determine what RPC call is altering your data. This is a read it once, you get the ah-ha why didn’t I think of that moment and you wont need to reference it again!
• [Temporary procedures : T-SQL]
  • We all know about temporary tables, but you can do that for stored procedures as well. Neat if you have a need for it
• [Conor vs. FOREIGN KEY join elimination]
  • Explains a bit of the history why a single column foreign key performs better than a multi-column FK. Its the old 80:20 rule at work, or probably more accurately 99.9:0.1 rule
• [SSRS -Report execution failed. Solution: SSPI=NTLM]
  • Seems downgrading to NTLM solves some integrated security issues if Kerberos is not playing nicely. Handy to know.
• [Take advantage of Table Variables NOT being transactional]
  • Ever needed to store state in a transaction that you didnt want to be rolled back? Think log tables! Read and learn
• [SQL SERVER – Fragmentation, Detection and how to resolve it]
  • Nice overview of fragmentation, and shows how to use DMV’s rather than the old school SQL 2000 DBCC approaches
• [CodePlex: SYS2 DMVs]
  • Hopefully this will serve as a good collection point – its very ‘young’ at the moment, but I would like it to live!
• [Locking improvements in SQL 2008 R2]
  • Handy to know about, unfortunately R2 has a price hike associated with it
• [Find the most expensive operations in Execution plans]
  • Handy way to take the XML plan and process it in a way to get a tabular list of the most expensive queries back. Very nice.

I suspect everyone who has dealt with scalar UDF functions in production environments are already VERY aware of the performance sucking capability they can have on your nice server. Here are some nice comments if you are not painfully aware:

• [Do Scalar UDFs give SQL Server a Bad Name?]
• [UDF Overhead – A simple example]

Gareth

So for all us who have spent time searching, or had pages pointed out to us , for options  this is an excellent new years present!

Many thanks to the SQLite guys for both their product, and their handy dandy search feature!

• [What's New in SQL Server 2008 R2 Editions]
  • Biggest first – Price increase! To say I’m disappointed is an understatement in this area. Specifically for developers  already have to support 2005 & 2008. Now they have to deal with R2, and persuade their customers to move to R2 if they want certain features.
  • MDS – Master Data Services – more on this one later. Pretty cool
  • Parallel Data Warehouse release
    • The DATAllegro purchase is now seeing the light as a Microsoft product!
    • Support of very large datasets (100Tb-1Pb). This is definitely something I would like to play with, but at $57,496 per processor…
  • Standard now has backup compression – but with the price tag…
• [SQL Server 2008 R2 November CTP – What’s New In Reporting Services?]
  • Report Builder 3.0 is available! Woohoo!
  • Enhanced sharepoint integration
  • Aggregates of aggregates
  • New Data Visualization Report Items
• [Using Filtered Statistics with Partitioned Tables]
  • Learn to appreciate filtered statistics!
• [MDS (Master Data Services) is x64 Only]
  • I totally get this, but this is the tipping point for me to repave my machines
  • With this requiring x64 and many new products requiring Windows 2008, Windows 7 or Vista it seems now is the time to move up.
• [MSSQL End of service dates for Service packs]
  • Support for SQL Server 2005 Service Pack 2 (SP2)  will end on January 12, 2010
  • Support for SQL Server 2008 RTM will end on April 13, 2010

More to come!

Gareth

[http://www.codersrevolution.com/index.cfm/2009/10/21/Sequoia-Voting-System-Witch-Hunt-err-Study-Project"]

The learning we can take away from this is if you don’t adequately cleanse then you can expect the data to become available! While its an interesting concept they apparently tried (and not too successfully) to do. The best way to clean a database is to create a new one and just copy in the data you want exposed. Don’t trust the handy dandy DROP/DELTE

If they wanted to expose/publish the 88 tables, then they should have created a new DB, copied in the tables and released it. Anything less than that you have to be VERY careful! And for the more security conscious it would be created on a recently wiped drive on a recently rebooted computer!

So looking at what SQLite3.exe has too offer it pretty much supports it out of the box. Very nice

Requirements:

• Loading speed
• Making the data to consuming applications available asap

While I love C# and frankly its hard to go back to C or C++, sometimes performance trumps the creature comforts we have become accustomed to.

Note: I did this without circling back to a C# implementation as I know the data and performance requirements  are tight and in this case I wanted max performance with no code! The biggest factor to a successful implementation is to ensure you use the tools best for the job, not just the ones you favor in that specific year.

So first things first – create a table to take the input

DROP TABLE IF EXISTS BookSales;
CREATE TABLE IF NOT EXISTS BookSales
(
   Store    int
  ,Date     varchar
  ,OrderReference varchar
  ,Line     int
  ,BookISBN varchar(14)
  ,Quantity int
  ,Price    int
, Primary Key (OrderReference,Line)
);

Next is the magic. We need to load the CSV into the table:

.separator "|"
.import BookSales.txt BookSales

Wow that was easy . You can see we set the separator to be a pipe rather than comma in this case, then the import.

.IMPORT [FileName] [Table]

Now the database is ready to be queried! But if we want to take it just one stage further:

.output SummaryBookSales.csv
SELECT Store, Date, BookISBN, SUM(Quantity), SUM(Price)
FROM BookSales
GROUP BY Store, Date, BookISBN;

Now we output the results of our simple aggregation into a pipe separated output file.

Tying this all together in a single configuration file, which we will call “BookAnalysisLoader.sql”, gives us:

DROP TABLE IF EXISTS BookSales;
CREATE TABLE IF NOT EXISTS BookSales
(
   Store    int
  ,Date     varchar
  ,OrderReference varchar
  ,Line     int
  ,BookISBN varchar(14)
  ,Quantity int
  ,Price    int
, Primary Key (OrderReference,Line)
);

.separator "|"
.import BookSales.txt BookSales

.output SummaryBookSales.csv
SELECT Store, Date, BookISBN, SUM(Quantity), SUM(Price)
FROM BookSales
GROUP BY Store, Date, BookISBN;
.exit

The last piece of the puzzle is the final execution:

sqlite3.exe BookSalesAnalysis.db3 < BookAnalysisLoader.sql

Now we have a newly created database with our analysis data in it, and we have a summary CSV file generated from the output. So we can load the CSV into Excel or another DB, or directly interrogate the DB for more analytical information – and all without coding!

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

• Improved query planner:
  • Through better use of statistics
  • Compile time option enables Analyze to better handle the index histograms
  • Additionally it was just plain improved as well!
• Recursive triggers
• Delete triggers fire during a REPLACE/MERGE
• More precise use of caching approaches with the
  • Shared Cache – basically cache the results within the application rather can on each thread. Now configurable on a per-thread basis rather than global.
  • Private – The old way

Well done to the SQLite team! Good stuff

More details can be found [SQLite Release 3.6.18 On 2009 Sep 11 (3.6.18)]

Definitely pretty cool stuff. Now to see how well it performs and verify it supports all the nice SQ L stuff  (sparse columns, PIVOT, stored procedures etc).

This will certainly be hyped by Microsoft over the coming weeks, and we will be trying it out! Much more to come on this topic,

Gareth

• Cumulative Update #3 for SQL Server 2008 SP1 (Build 10.00.2723)
• Cumulative Update #6 for SQL Server 2008 RTM

Things that stand out are:

• [970399]  FIX: The MAXDOP option for a running query or the max degree of parallelism option for the sp_configure stored procedure does not work in SQL Server 2008
• [969844] FIX: You receive inconsistent results when you run index-related DMVs to return statistical information about missing indexes in SQL Server 2005 or in SQL Server 2008
• [969997] FIX: You receive an incorrect result when you query data from a linked server that is created by using an index OLE DB provider in SQL Server 2005 or in SQL Server 2008
• [970507] FIX: Error message in SQL Server 2008 when you run an INSERT SELECT statement on a table: “Violation of PRIMARY KEY constraint ‘<PrimaryKey>’. Cannot insert duplicate key in object ‘<TableName>’”
• [971064] FIX: Quotation marks are rendered incorrectly when you export a SQL Server 2008 Reporting Services report to a .csv file

Obviously you need to really read to understand if you have seen any of the problems the fixes address to be fully aware – and definitely don’t apply unless you need a fix!

Gareth

Microsoft is updating the branding for SQL Services and SQL Data Services. Effective immediately, SQL Services will be called Microsoft SQL Azure, and SQL Data Services will be Microsoft SQL Azure Database.

There will be more news and updates related to our Software + Services strategy next week at Microsoft’s Worldwide Partner Conference in New Orleans. For those of you not attending in person, all the news and information can be found at: http://www.digitalwpc.com/ . Check back often starting next Monday, July 13, for daily updates.

More about TSQL Support in SQL Data Services can be found here. Another announcement of this can be found at “SQL Azure – Same great platform, just a better aligned name”

Apparently its a ‘Cloudy’ week

Gareth

The first thing to do is go and grab the latest source! The source code is currently hosted on SourceForge at: http://sourceforge.net/projects/sqlite-dotnet2/. If you go to this page you there are links (or there were at the time of writing!!) to “Download Now!” or “View all files“. If you are looking for the source you need to chose “View all Files”. From here you should choose the latest source package (currently “SQLite-1.0.63.0-source.zip”) and download and unzip to your drive. You should now have the latest and greatest ‘stable’ source code, if you need to pick up an in flight fix or need to most up to date you may have to use CVS to download the package. Unfortunately (albeit fortunate for this article) “SQLite-1.0.63.0-source.zip” contains an issue where the build didnt completely work straight out of the box for Win32 builds. I contacted the team through the developer forum (http://sqlite.phxsoftware.com/forums/) and the problem was resolved in the latest CVS code patch (Patchset 415). So given this we need to pull it from CVS rather than the prepackaged code. NOTE: Normally speaking you should only need (and I would recommend only) the packaged code rather than CVS. Only if you want to live on the bleeding edge, or have a real need to get the latest release should you really use the CVS version. If fact the team doesnt even recommend using the CVS version (http://sqlite.phxsoftware.com/forums/t/1814.aspx).

However to complete the full circle and want to go down the CVS route and you are not familiar with CVS you can find out more information here. Raw CVS executables for Windows can be found here, but I would recommend if you are going to make a habit out of it you get and use TortoiseCVS instead.

cvs -d:pserver:anonymous@sqlite-dotnet2.cvs.sourceforge.net:/cvsroot/sqlite-dotnet2 login
cvs -z3 -d:pserver:anonymous@sqlite-dotnet2.cvs.sourceforge.net:/cvsroot/sqlite-dotnet2 co -P SQLite.NET

[Note if the first line fails create an empty ".cvspass" file and retry it]

Now you should have the latest code straight from the CVS repository, or if you are more sensible the packaged ZIP file covered at the beginning of the article.

Now on to the building! In the top level folder there are 2 interesting/key files:

• readme.htm – This details the latest release notes associated with the provider code, and covers the differences between the various historical releases.
• SQLite.NET.sln – This is our well loved and trusted VS2008 solution file!

Starting the solution file the solution loads up and with the latest version I’m using8 projects are loaded:

• SQLite.Interop
• System.Data.SQLite – Compact
• System.Data.SQLite – Managed Only
• System.Data.SQLite – Netmodule
• System.Data.SQLite.Linq
• test
• testce
• testlinq

Out of the box you should be able to build the release and debug files. You will also notice slightly different icons associated with the projects “System.Data.SQLite – Compact” & “testce”. This is because these builds are targeting a mobile or Compact framework device.  They dont hurt to be there so feel free to ignore them as they wont be used unless you a compact/mobile build type.

Congratulations you should have successfully built the components and can verify your build by running the “test” application.

Now on to the more technical deep dive of how it works

Firstly lets be clear – this is certainly a non-trivial build process! This is merging C code & .Net code into the same binary DLL using netmodules. More on netmodules can be found at “How to: Build a Multifile Assembly“. Frankly I’m going to just dive into a little more detail here as this concept is key to the build.

Purely looking at the project in Visual Studio is a little misleading:

Hmmm – a “Windows Application” – really…. However looking at the “System.Data.SQLite – netmodule.csproj” in a text viewer you see something very different:

<OutputType>Module</OutputType>

Obviously this is very different than “<OutputType>Library</OutputType>”, “<OutputType>WinExe</OutputType>”, “<OutputType>Exe</OutputType>”. This is the netmodule magic we covered above – all this is largely hidden from the user if they just look at the VS interface. This magic is then referenced through the SQLite.Interop.vcproj though:

<Tool
Name="VCLinkerTool"
AdditionalDependencies="..\System.Data.SQLite\bin\System.Data.SQLite.netmodule"
OutputFile="../bin/System.Data.SQLite.DLL"
ModuleDefinitionFile="src\sqlite3.def"
EmbedManagedResourceFile=""
DelayLoadDLLs="advapi32.dll"
RandomizedBaseAddress="1"
DataExecutionPrevention="0"
ImportLibrary=""
TargetMachine="5"
KeyFile="..\System.Data.SQLite\System.Data.SQLite.snk"
CLRUnmanagedCodeCheck="true"
/>

The good news about this is that this is just a ‘dependency’ and is fully supported in the C/C++ interface in VS2008. A little more can be read about here “.netmodule Files as Linker Input”

Alright so we now have the background knowledge to understand how this build is actually being used. So now lets look at the what the build sets up for us (the below is really the Win32/x64 build):

• Debug
  • SQLite.Interop
  • System.Data.SQLite – Managed Only
  • System.Data.SQLite.Linq
  • test
  • testlinq
• Debug – Stock
  • System.Data.SQLite – Managed Only
  • System.Data.SQLite.Linq
  • test
• Release
  • SQLite.Interop
  • System.Data.SQLite – Netmodule
  • System.Data.SQLite.Linq
  • test
• Release – Stock
  • System.Data.SQLite – Managed Only
  • System.Data.SQLite.Linq
  • test

The project dependencies are:

• SQLite.InterOp -> System.Data.SQLite – netmodule
• System.Data.SQLite.Linq -> SQLite.InterOp
• Test -> SQLite.InterOp

You can see that only the “Release” build doesn’t include the “System.Data.SQLite – Managed Only” version. Any build other than this one generates a build that requires both “System.Data.SQLite” and “SQLite.Interop.DLL”. The “Managed Only” isn’t a pure managed only version of SQLite, it still uses the native SQL C runtime code – but it makes the call out to the Interop dll. The ‘netmodule’ version is used to embed in the ADO.NET C# classes into the ‘System.Data.SQLite’ DLL, all other builds of ‘System.Data.SQLite’ just reference the interop DLL.

Release Build

As mentioned before, in the release build SQLite.Interop is where all the magic happens in this build! This is the one that outputs bin\System.Data.SQLite.DLL (note in all other builds this generates ‘SQLite.Interop.dll’ or in the stock case ‘sqlite3.dll’). However since this is really compiling the standard C SQLite code there is a little more magic going on here. This project has the linker dependency on “..\System.Data.SQLite\bin\System.Data.SQLite.netmodule”. This is how we get the C# ADO.NET provider injected into the System.Data.SQLite.DLL through the netmodules. However this process has the side effect of not being able to debug into the provider . So for source code debugging use the debug build that references “Managed Only” rather than the netmodule.

Debug Build

From now on it is fairly standard, although I have seen an odd quirk is that if you attempt to debug into the regular “Debug Build” an error is through regarding the “‘OutputType’ failed”, if you use the “Debug – Stock” build this goes away as the ‘netmodule’ is not included in the build.

Happy compiling and debugging!

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

Most Simple – Data Source
Format: “Data Source=[DBFileName]”
Example: @”Data Source=C:\CSharpHackerDemo.db3″

This connection string will create the database if it doesnt already exist without notifying you. For some utility functions this may be exactly what is desired, in some cases you want to know your DB doesnt exist!

If you want to programatically create the file intentionally, presumably if  “!File.Exists()” you can do:

SQLiteConnection.CreateFile(@"C:\CSharpHackerDemo.db3");

Only Open if it exists – FailIfMissing
Format: “Data Source=[DBFileName];FailIfMissing=True;”
Example: @”Data Source=C:\CSharpHackerDemo.db3;FailIfMissing=True;”

With this connection string if the DB file is not present the open will fail.

Secure/Encrypt the DB – Password
Format: “Data Source=[DBFileName];Password=[123456789];”
Example: @”Data Source=C:\CSharpHackerDemo.db3;Password=[123456789];”

With this connection string if the DB file will be created or opened using the password as the encryption key.

Other options:

• ReadOnly – When enabled, the database will be opened for read-only access and writing will be disabled. Default False.
• CacheSize – Sets the cache size for the connection. Default value is 2000 bytes. Under the covers this calls “PRAGMA cache_size={0}”.
• PageSize – Sets the database page size. Under the covers this calls “PRAGMA page_size={0}”. Default is 1024 bytes.
• MaxPageCount – Sets the maximum number of pages the database may hold. Default is 0 which is unrestricted. Under the covers this calls “PRAGMA max_page_count={0}”.
• DateTimeFormat - Sets the datetime format for the connection. Default is ISO8601
  • Ticks - Using ticks is not recommended and is not well supported with LINQ.
  • ISO8601 - Default format for this provider.
  • JulianDay - JulianDay format, which is what SQLite uses internally
• JournalMode - Determines how SQLite handles the transaction journal file.  Under the covers this calls “PRAGMA journal_mode={0}”. Default “Delete”
  • Delete - default mode, this causes SQLite to create and destroy the journal file as-needed.
  • Persist - When this is set, SQLite will keep the journal file even after a transaction has completed.  It’s contents will be erased, and the journal re-used as often as needed.  If it is deleted, it will be recreated the next time it is needed.
  • Off - This option disables the rollback journal entirely.  Interrupted transactions or a program crash can cause database corruption in this mode!
• DefaultIsolationLevel - Sets the default isolation level for transactions on the connection. Default “Serializable”. Note this is a standard .Net “System.Data.IsolationLevel” type. Do not even try to change this . Only “Serializable” and “ReadCommitted” can be used, all others will throw a “NotSupportedException”. However for completeness the full list and meanings are outlined below:
  • Serializable - A range lock is placed on the DataSet, preventing other users from updating or inserting rows into the dataset until the transaction is complete.
  • ReadUncommitted – A dirty read is possible, meaning that no shared locks are issued and no exclusive locks are honored.
  • ReadCommitted - Shared locks are held while the data is being read to avoid dirty reads, but the data can be changed before the end of the transaction, resulting in non-repeatable reads or phantom data.
  • RepeatableRead - Locks are placed on all data that is used in a query, preventing other users from updating the data. Prevents non-repeatable reads but phantom rows are still possible.
  • Snapshot - Reduces blocking by storing a version of data that one application can read while another is modifying the same data. Indicates that from one transaction you cannot see changes made in other transactions, even if you requery.
  • Chaos - The pending changes from more highly isolated transactions cannot be overwritten.
  • Unspecified - A different isolation level than the one specified is being used, but the level cannot be determined.
• Version – Sets the default version of the SQLite engine to instantiate.  Currently the only valid value is 3, indicating version 3 of the SQLite library.
• Synchronous – Sets the synchronization mode (file flushing) of the connection string.  Default is “Normal”.  Under the covers this calls “PRAGMA synchronous={0}”. Supported values are:
  • Normal – Normal file flushing at critical sections of the code
  • Full – Full file flushing after every write operation
  • Off – Use the default operating system’s file flushing, SQLite does not explicitly flush the file buffers after writing
• UseUTF16Encoding – Boolean indicator setting the encoding for the connection.  The default is “False” which indicates UTF-8 encoding.
• Pooling – Sets whether or not to use connection pooling.  The default is “False”
• BinaryGUID - Sets to or not to store GUID’s in binary format.  The default is True which saves space in the database.
• Deprecated Uri – depreciated call to the data source property.
• DefaultTimeout – sets the default command timeout for newly-created commands.  This is especially useful for commands used internally such as inside a SQLiteTransaction, where setting the timeout is not possible. Time is in seconds, the default is 30 seconds.
• Enlist – Determines whether or not the connection will automatically participate in the current distributed transaction (if one exists). Default is True.
• LegacyFormat – If enabled, uses the legacy 3.xx format for maximum compatibility, but results in larger database sizes. Default is False. Under the covers this calls “PRAGMA legacy_file_format={0}”.

So this is the connection string options available on the latest release. The interesting part is how I found this – in the source code (so at least now you dont need to do that part). However the next blog will be how to build the provider to help you diagnose and understand any issues you come across.

Series Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

 SQLiteConnection.CreateFile("c:\\mydatabasefile.db3"); SQLiteConnection.CreateFile("c:\\mydatabasefile.db3");

So starting with the biggies:

1) Limited Database enforced Type Safety:

CREATE TABLE  IF NOT EXISTS Book
(
BookISBN     varchar(14) primary key
,Name         varchar(200)
,PublisherId    int
);

INSERT INTO BOOK(BookISBN,Name)
VALUES(
     "ReallyLongValueThatShouldCauseAProblem"
    , "CSharpHacker SQLite Manual");

Most DBs honor the restrictions of the character lengths. SQLite will record the information you submit to it – even it is technically breaks your DDL definitions. More about this can be read about it Datatypes In SQLite Version 3. What should specifically be called out is:

Any column in a version 3 database, except an INTEGER PRIMARY KEY column, may be used to store any type of value.

So a more outlandish example would be:

DROP TABLE IF EXISTS Issue;
CREATE TABLE  IF NOT EXISTS Issue
(
 IssueId 	int
,Description	varchar(200)
);
INSERT INTO Issue(IssueId,Description)
VALUES(
     "ReallyLongTextValueThatShouldCauseAProblem"
     , "CSharpHacker SQLite Manual");
SELECT * from Issue;

If you run the above you get “ReallyLongTextValueThatShouldCauseAProblem|CSharpHacker SQLite Manual”, with no errors!
Is that the sound of DBA’s running away screaming in horror…? Believe it or not this was intentional, and ‘sometimes’ sensible . I’ll cover a little more on that later, but lets go full circle on this.

Underneath the covers there is a 64 bit row identifier attached to every row, but if you want to use your own you can override that to use your own values. If you do this then this column categorically has to be an INTEGER value. So looking at the SQL below you can see we have now changed the type from ‘int’ to ‘INTEGER PRIMARY KEY’. This now enforces you have a value that is an integer for this one column, and the below fails with a “SQL error near line xx: datatype mismatch”.

DROP TABLE IF EXISTS Issue2;
CREATE TABLE  IF NOT EXISTS Issue2
(
 IssueId 	INTEGER Primary key
,Description	varchar(200)
);
INSERT INTO Issue2(IssueId,Description)
VALUES(
     "ReallyLongTextValueThatShouldCauseAProblem"
    , "CSharpHacker SQLite Manual");
SELECT * from Issue2;

So one final warning “INTEGER PRIMARY KEY” is the same as “integer primary key”, but not “int primary key”. If you are wanting to override the internal 64 bit row identifier you have to use “integer” and not “int”, “bigint”, “short integer”. Thats an easy one to miss! Final quirk is that the “INTEGER PRIMARY KEY” needs to be exactly that or “INTEGER PRIMARY KEY ASC” (case insensitive), but “INTEGER PRIMARY KEY DESC” will not use the internal ID.

2. Only one process can write to the DB at the same time

The locking in SQLite is not fine grained! If anyone is writing to the DB it locks everyone and everything else out of the DB. All readers and writers are blocked until the pending write is done. So if you are expecting a lot of concurrent writes to the database that may need to be interleaved with some reads from other threads/processes you may experience some blocking. In reality it shouldnt matter too much as the writes are normally speedy. It should also be noted that in SQLite v2 there was a possibility of writer starvation where so many reads are going on the writer was never able to lock the file to perform the write – this has been addressed in SQLite V3.

3. Limited support for ALTER TABLE

Now I bet this has driven off the DBA’s now! Because of its heritage SQLite team hasn’t ever needed to support the more complex data migrations supported by other database products. You can use to add a column to the end of a table or to change the name of a table. However if you want to make more complex changes in the structure of a table, you have to recreate a new table and then copy the contents of your old table into the new one.

4. Foreign Keys are ‘recorded’ but not enforced

Technically the same thing can be achieved using the ‘Triggers’, but you need to be aware that these are ‘soft RI’ keys and are not enforced by the database engine.

5. Support for UPSERT via “REPLACE” or “INSERT OR REPLACE”

For MySQL users they are familiar with REPLACE, SQLite has aliased REPLACE to its native “INSERT OR REPLACE’.

So if you are still reading that is good! This is definitely the scary page that drives a few DBA’s into fits, but its sensible to know what you are using and its behaviors before it surprises you.

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

However back to the question – How does it stack up?

As you can see SQL Server definitely has the most features, in fact as far as features go its a slam dunk. However often feature count just isn’t everything. The major drawback for SQL Server is that it requires a Server process to run, and that then means maintenance, probably a DBA and all of a sudden things have got complicated (and expensive)! So we come back to the question “What are we looking to do with the database?”. If the answer is heavy duty processing that requires stored procedures, cubes and large amount of concurrent remote access by users – SQL Server is definitely your answer. However I have to think that you are reading this because SQL Server is too heavy weight for your needs. So on to the other candidates.

SQL CE is a Microsoft database that started in the mobile space and moved up to the PC platform. Overall is has reasonable features but is known to be slower and heavier than SQLite. It definitely has some strengths in the SQL Server ==> SQL CE replication, but outside of that its largely even or less featured that SQLite. If you are looking for replication definitely checkout SQL CE, but I would also say the Microsoft Synch framework is evolving and we may see a native SQLite provider (or determine that the ADO.NET is sufficient) in this area.

SQLite is a lightweight add-on to C# (System.Data.SQLite.DLL ~ 850Kb). You may note this is heavier than the often referenced SQLite runtimes take “200Kb”, but in this 850Kb (457Kb Zipped) is the SQL DLL, ADO.NET SQLite provider and the magic glue to bring these two worlds together. If you are using .Net you can assume you have to assume your package size will go up by 457Kb, and the runtime size on disk will be ~850Kb. Performance of SQLite ranks it above SQL CE for insert tests, and it has a lower file size – and the database can be assessed on a massive number of operating systems!

Before you jump whole heartedly on the SQLite bandwagon, it is only fair to also point out SQLite has some quirks (SQLite folks call these ‘features’ – I’m not going to argue, but its approach is different in a couple of cases). I’m going to cover these in the next blog entry “Part 5″. These are things that you definitely need to be aware of rather than discover!

The key question that the article should be asking is “What type of application features benefits from SQLite?”. The best way I can describe SQLite is that is is an “Application Data Store” that you then have ANSI-92 SQL access to. This removes the needs of rolling your own indexing into files, creating convoluted file system ‘databases’ etc. If you can wrap your head around how you can use SQLite to your advantage I think you will be pleasantly surprised.

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

• Mono on the iPhone…. To say this is exciting is an understatement. Unfortunately I’m a little afraid that Apple wont like it too much (read allow it in their TOS).
  

  “3.3.2 An Application may not itself install or launch other executable code by any means, including without limitation through the use of a plug-in architecture, calling other frameworks, other APIs or otherwise. No interpreted code may be downloaded and used in an Application except for code that is interpreted and run by Apple’s Published APIs and built-in interpreter(s)”

  .Mono is normally an interpreted language very similar to Java and the TOS explictly disallows running byte code on the machine.  So this could be VERY interesting, or it could be a nice concept and not never to hit the Apple store. Definitely watch this space!Otherwise we will end up with a compiler than generates Object C code

• Firefox 3.5 should be out tomorrow.
• Another EDW appliance comes into the fold with Aster Data.
• Simple problems highlighted with “SQL Server 2005 reporting in management studio“.

Gareth

In order for the code to run you will need to have installed “SQLite.Net” and reference the “System.Data.SQLite” DLL (probably found at "C:\Program Files\SQLite.NET\bin\System.Data.SQLite.dll"). With this and the below code you should have something to get started with.

using System;
using System.Data;
using System.Data.Common;
using System.Data.SQLite;

namespace SampleSqliteConsoleDemo
{
   class Program
   {
      static void Main(string[] args)
      {
         // Create table and information using native types
         using (SQLiteConnection conn = new SQLiteConnection(@"Data Source=c:\testperf.db"))
         {
            conn.Open();
            using (SQLiteCommand comm = new SQLiteCommand())
            {
               int limit = 100;
               comm.Connection = conn;
               comm.CommandText = "DROP TABLE IF EXISTS test;" +
                                  "CREATE TABLE IF NOT EXISTS test (n integer);";
               try
               {
                  comm.ExecuteNonQuery();
                  using (SQLiteTransaction tran = conn.BeginTransaction())
                  {
                     // You can use the below in the for loop if you 'need'
                     // to, and it does make it 'easier' to read as you dont
                     // need the additional param code but not only is this
                     // faster, but avoids the potential for SQL injection.
                     //comm.CommandText = "INSERT INTO test values (" + i.ToString() + ")";

                     comm.CommandText = "INSERT INTO test values (@myTestIndex)";
                     DbParameter param = comm.CreateParameter();
                     param.ParameterName = "myTestIndex";
                     param.DbType = DbType.Int32;
                     comm.Parameters.Add(param);

                     for (int i = 0; i < limit; i++)
                     {
                        param.Value = i;
                        comm.ExecuteNonQuery();
                     }
                     tran.Commit();
                  }
               }
               catch (SQLiteException exception)
               {
                  Console.WriteLine("Failed :" + exception.Message);
               }
            }
            conn.Close();
         }

         // Access data using the more traditional ADO.NET namespace
         DbProviderFactory fact = DbProviderFactories.GetFactory("System.Data.SQLite");
         using (DbConnection cnn = fact.CreateConnection())
         {
            cnn.ConnectionString = "Data Source=c:\\testperf.db";
            cnn.Open();
            {
               IDbCommand dbcmd = cnn.CreateCommand();
               string sql =
                  "SELECT n " +
                  "FROM test";
               dbcmd.CommandText = sql;
               IDataReader reader = dbcmd.ExecuteReader();
               while (reader.Read())
               {
                  Console.WriteLine("n: " + reader.GetInt32(0));
               }
           }
         }
      }
   }
}

So the things to note are:

• This this is obviously not a guide for great programming – but it should touch on the highlevels to access information from SQLite.
• We are using a combination of direct instantiation of “SQLiteConnection” and the ADO.NET base classes. Normally you should try to just use the base classes where possible for portability (should you want to change the DB). We chose both just to highlight the two approaches.
• The DDL creation uses the SQLite “IF EXISTS” for DROP TABLE and “IF NOT EXISTS” for the CREATE TABLE. These are handy features, but are not standard.
• Instead of using the commonly found string concatenation approach to execute the SQL (most introduction web examples show this) we show the preferred approach using DbParameter. This is not only safer (avoids SQL Injection), but is faster as well! Hard to complain about faster and more secure!

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

UPDATE 2009-06-29: It seems there are similar issues with SQL2005 as well, have a look SQL Server version has a known compatibility issues with Windows 7 RC. This warning is genuine and applies to all SQL Server editions, including all Express editions.

Gareth

Using the native tool – SQLite3.exe

Unfortunately while the package “http://sourceforge.net/projects/sqlite-dotnet2” does contain the SQLite runtime access DLL’s it doesn’t contain all the tools that are supplied as part of the base package. The most useful one is SQLite3, this can be found on the SQLite download page. Search for “Precompiled Binaries For Windows” and find the SQLite_x_y_z.zip download, at the time of writing this is the current version sqlite-3_6_15.zip (note if you are reading this at some point later than 24 June I suggest you ensure you get the latest version).

SQLite3.exe provides command line access to the SQLite engine, for full usage and capabilities check out http://www.sqlite.org/sqlite.html for more information.

However to create a very (and I really do mean VERY simplistic) database we can write the SQL schema and then add it into a new database.  So lets quickly knock up a very simple example:

CREATE TABLE Publisher
(
PublisherId     int primary key
,Name         varchar(200)
);
CREATE TABLE Book
(
BookISBN     varchar(14) primary key
,Name         varchar(200)
,PublisherId    int
);

INSERT INTO BOOK(BookISBN,Name) VALUES(“1-84356-028-3″, “CSharpHacker SQLite Manual”);
INSERT INTO BOOK(BookISBN,Name) VALUES(“1-84356-029-3″, “SQLite vs VistaDB”);
INSERT INTO BOOK(BookISBN,Name) VALUES(“1-84356-030-3″, “SQLite vs SQL Server CE”);
INSERT INTO BOOK(BookISBN,Name) VALUES(“1-84356-031-3″, “SQLite vs Velocity Caching”);
.quit

This is our super simple schema that will evolve as this series moves on, but say for now we think this is the best schema since sliced bread was invented! You will notice the .quit at the end, this is used because we are just going to pipe this into the Sqlite3 application to create the database. So you need to save the above code into a file “CSharpRamblingCoder.sql” and then run the below:

sqlite3.exe ramblingCoder.db3 < CSharpRamblingCoder.sql

You will notice this will auto create the database file and fill it up for you. In fact one of the quirkier aspects of this database is that the default behavior is that it will auto create the data file if it cant find the one you specified. For example if you use SQLite3 or the .Net provider and supply a DB file that doesn’t exist then that DB container file gets created on the fly. We will go into a bit more detail on the SQL constructs SQLite uses in a later series, but for now I think this covers the initial “How do I create a new SQLite database?” and “How do I get my schema into a new SQLite database?”

Users of Visual Studio 2008 Full Edition

So assuming you followed the instructions in Part 1 and selected the full Visual Studio 2008 add-in you have a second way to create, setup and interrogate the database. This has to do with the “Server Explorer”, note at the time of writing this is an early cut of code so just be aware of that .

So if you now look at the “Server Explorer” you should see something like this (or have previous connections setup):

From here if you right mouse button on “Data Connections” you will be prompted to add a “New Connection…”:

This then will bring you to the “Add Connection” dialog, ensure that the “Data source:” is setup to be “SQLite Database File …” rather than anything else.

If this is not the case you need to “Change…” and select the SQLite driver:

Now make sure you have filled in the Database file name as a pathed location.

And congratulations you have just made an empty SQLite database!!!

To confirm this open up the CSharpCoderSample and see that you have holding places for Tables, System Tables and views.

Having a DB without any tables or functionality obviously isnt terribly useful, so lets add a table via the UI. Right mouse on the “tables” and select “Add New Table”.

This opens up a new tab in Visual Studio and you can start to fill in the information as you would normally:

Here is a new table I just imagined I may need at some point in the future

From here you save the table, I prefer Control+S but any save option works and you are prompted for the table name.

Ok we now have our first graphically built table – woo hoo!

So to expedite things you can run:

sqlite3.exe c:\CSharpCoderSample.db3 < CSharpRamblingCoder.sql

And after pressing refresh you can see the tables updated in the Server explorer.

Note since this add in is a early release it is not as fully featured as you would ideally like, but it is a nice way to peer into the database structure and actual data without needing to leave the IDE which is always a nice to have!

So now we know how to create a empty database & add schema. The final part is how to delete the database! Thankfully this is equally as obvious , delete the file c:\CSharpCoderSample.db3.

Now we are ready to get coding! Part 3 covers our first C# application to utilize the SQLite database.

Related Links:

• SQLite for C# – Part 1 – Am I allowed to use it?
• SQLite for C# – Part 2 – How do I setup a SQLite DB (without coding)
• SQLite for C# – Part 3 – My first C# app using SQLite aka Hello World
• SQLite for C# – Part 4 – So how does SQLite stack up against other DB’s?
• SQLite for C# – Part 5 – SQLite ‘features’, or ‘quirks’
• SQLite for C# – Part 6 – SQLite Connection String Definitions
• SQLite for C# – Part 7 – Building SQLite.Net from source
• SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

Some may consider this a non-event, while it may seriously turn the world upside down for others. The latest set of Oracle drivers can be found here http://www.oracle.com/technology/tech/windows/odpnet/index.html

The Oracle Data Provider for .NET (ODP.NET) features optimized ADO.NET data access to the Oracle database. ODP.NET allows developers to take advantage of advanced Oracle database functionality, including Real Application Clusters, XML DB, and advanced security. The data provider can be used with the latest .NET Framework 3.5 version.

ODP.NET makes using Oracle from .NET more flexible, faster, and more stable. ODP.NET includes many features not available from other .NET drivers, including a native XML data type, array parameters, RAC optimizations, and statement caching. ODP.NET is designed for scalable enterprise Windows solutions by providing full support for Unicode and local and distributed transactions. Distributed transactions are supported using the Oracle Services for MTS.

Frankly this package ranks as #1 & #2 when using Google to search for “oracle .net provider” so I suspect a lot of developers who use Google will have gone straight to this approach anyway.

Update

It seems this story is gathering momentum now:

• http://cwebbbi.spaces.live.com/Blog/cns!7B84B0F2C239489A!4573.entry
• http://www.theregister.co.uk/2009/06/18/microsoft_kills_oracle_connector/

Gareth