Comments on: SQLite for C# – Part 8 – Loading CSV/Pipe into SQLite via command line

By: Gareth

Gareth — Fri, 06 Aug 2010 03:40:04 +0000

Well I tried a quick test of this (rather that diving into the code!). My test was to generate a C# program to quickly create a 342Mb file consisting of 1 int, and 3 text fields in a CSV format. It generated 2,000,000 records for the test. This was then loaded using the “.import” approach outlined above and it ran fine (and quickly). It took approximately 20 seconds to generate a 421Mb SQLite DB and during this load the memory size of sqlite3 remained level and didn’t increase during the load phase.

Hopefully this helps some, if you want my test files I can supply them – but on the surface there doesn’t seem to be an issue here.

Gareth

By: Gareth

Gareth — Tue, 03 Aug 2010 21:45:40 +0000

I have not tried a very large file (of that size), but I seem to remember looking at the code a while a go and saw that it loaded the files in chunks – so that shouldn’t be the issue (or perhaps that us my mind playing tricks on me!). However I’ve been using notepad for the generation of the files and it loads them without issue. Does it work on a smaller file? I presume you are using the .import?

By: Breck

Breck — Tue, 03 Aug 2010 20:29:57 +0000

Have you tried importing of a very large file? Say on order of 200MB+. I tried it with the native SQLite3.exe and if failed without any messages. Turns out after I looked at the C implementation, the developer implemented the parsing of the file such that it never finds the newlines. I am also not sure if (did not spend enough time looking at the code, to see if there was a memory or file size issue. It appears it may attempt to load the whole file at once which defeats one of the needs I have.

By: Gareth

Gareth — Tue, 11 May 2010 12:58:51 +0000

These are really built into the SQLite3 application rather than the OLE-DB driver. So the simple answer is no, however if you look at the C# implementation they have ported the whole thing including the main program, via translation – so dont expect the code to be pretty, to C#. So you could lever that, or invoke the native SQLite3 application. It all depends on what you are trying to achieve!

Gareth

By: goose1_fr

goose1_fr — Mon, 10 May 2010 14:12:00 +0000

is it possible to implement this special command (.mod ; .separator and .import …) in C# program ?

By: Gab

Gab — Sat, 08 May 2010 12:18:51 +0000

Simple, Ordered, Fully Explained. Excelent Tutorial Congrats. And thanks you. I’ll use it today, u’ve saved me to use MS Access !! IIUGGHHH !!! Thansk a lot.

Gab.

By: Sean

Sean — Thu, 22 Apr 2010 03:45:13 +0000

Just wanted to say thanks for the excellent series. I’ve read it 2 or 3 times now!

By: Sean

Sean — Sat, 23 Jan 2010 20:39:03 +0000

A minor update – I haven’t tried it yet, but some recent tweaking in the dev branch is yielding improved perf for the .Net port. They are still trying to fix bugs though….

By: Gareth

Gareth — Tue, 27 Oct 2009 00:40:37 +0000

I’ve only downloaded it enough to see how well the translation went. Got to say I dont like the generated code too much. Outside of that I’ve not had much to do with it. I would like to really see a native version as that avoids the whole 64 bit vs 32 bit question, and I would hope it would also help with performance (although the charp-sqlite doesnt reflect any performance gains). My hope is that the Mono team include it, or MS includes it as a native implementation

By: Sean

Sean — Wed, 14 Oct 2009 14:04:51 +0000

Have you played with the .Net port of SQLite? http://code.google.com/p/csharp-sqlite/

I’d be interested in knowing if you noticed any differences…

By: Jenny

Jenny — Fri, 09 Oct 2009 11:13:48 +0000

Thank you! for the very informative and useful writeup. Even those Wrox and Wiley’s heavy and expensive books couldn’t cleared what you have done in few lines.

By: Gareth

Gareth — Thu, 24 Sep 2009 11:56:15 +0000

Glad to have helped! There is information out there, the trick is finding it!

By: Raimis

Raimis — Wed, 23 Sep 2009 12:30:03 +0000

Thanks

You helped me to understand this a little bit better. Too bad there are not much information about c# SQLite around