Topic: Import and Export

Dmitry has posted an import export script in FAQ category of MVD forums.


I wanted to add a bit of user control and friendliness to it, such as ability to specify file name, location etc via windows open and save dialogs. I think I have done this part (hopefully correctly) thanks to Dmitry's kind help providing such script earlier.


Three issues I have faced and couldn't resolve:
1. I wanted include multiple tables data (in this case 2) in export and import to build an understanding of exporting and importing the whole database, but couldn't get it working?. More often than not database file will have multiple tables.


2. I wanted to add duplicate counts on info message (after import process finish) when importing with duplicate check, but no idea how to do it. I put a comment in the script too about it.


3. Adding a dialog with progress bar which would be displayed during import export process running. This would be very helpful when exporting and importing larger databases.


Could anybody have a look at the sample project attached here and implement the above mentioned features and correct my mistakes if there is any please?

Post's attachments

Attachment icon Import-Export Dmitry - Work.zip 34.14 kb, 479 downloads since 2018-01-27 

Adam
God... please help me become the person my dog thinks I am.

2 (edited by mathmathou 2018-01-27 20:32:21)

Re: Import and Export

Hello AD,


This is a quick and "disrty" answer because I'm soon leaving home for a motorcycle ride with friends (Sunday morning here).


For the first part of your question : exporting multiple query to 1 csv file. This can not work the way you do it because the CSV file you save to is created for each query, the last one overwriting the previous one, thus you only have part of the result in your csv file.


One workaround would be to pass to your export procedure, not a Filename, but a TStringList.
This list is created BEFORE calling the export procedure, thus each results are appended to it, and destroyed after the export. This way, all data of all queries are in the StringList and saved to the same file.


I made some changes to your code as well :
- I deleted the image field from the second query because the result was ugly in the csv file (but you can add it back if needed)
- I hard-coded the export filename because I lacked time to do it properly.


The whole process is now
1 - Create the StringList
2 - Call export procedure that append it's results to the StringList as many times as needed
3 - Save the StringList to file
4 - Cleanup everything and end.


Export Procedure :

Procedure ExportToSl (SQL: string; Sl: TstringList);
var
Results: TDataSet;
i,c: integer;
sValue, sValues: string;

begin
SQLQuery(sql, Results);
c := Results.FieldCount-1;

  while not Results.eof do
  begin
  sValues := '';
  for i := 0 to c do
      begin
      sValue := Results.Fields.Fields[i].AsString;
      if (Results.Fields.Fields[i].DataType = ftFloat) or (Results.Fields.Fields[i].DataType = ftLargeint) then
          sValue := ReplaceStr(sValue, ',', '.')
          else
          begin
          sValue := ReplaceStr(sValue, ',', ',');
          sValue := ReplaceStr(sValue, #13#10, '|');
          sValue := '''' + escape_special_characters(sValue) + '''';
          end;

          sValues := sValues + sValue+',';
        end;

  if sValues <> '' then SetLength(sValues, Length(sValues)-1);
  sl.Add(sValues);
  Results.next;
  end;
Results.Close;
end;

As you can see, the StringList is passed as parameter, just like you did wit the filename, but not created in it. It's the calling procedure that will create and free it, like thi :

procedure Form1_bExport_OnClick (Sender: string; var Cancel: boolean);
var
    Sl : TstringList;
begin
Sl := TstringList.Create;
sFileName := 'ExportedData-' + FormatDateTime('dd-mm-yyyy hh-nn-ss', now)+'.csv';
SaveDialog := TSaveDialog.Create(Form1);
SaveDialog.FileName := sFileName;
SaveDialog.DefaultExt := 'csv';
SaveDialog.Filter := 'Database|*.csv|Any file|*.*';
SaveDialog.Options := ofOverwritePrompt+ofHideReadOnly+ofEnableSizing;

if SaveDialog.Execute then
   begin
       ExportToSl('SELECT lastname, firstname, salary, dateofbirth, isSmoke, comment FROM employees', Sl);
       ExportToSl('SELECT coDate,  coName, coActive FROM Company', Sl);
       Sl.SavetoFile('.\export.csv');

       ShowMessage('Data has been exported in to the specified folder');
   end;
SaveDialog.Free;
Sl.Free;
en;

This way, you can call the ExportToSl procedure as many times as you want, on as many queries as you want, everything is added to the stringlist (appended) and saved in one shot.


Hope this helps, I leave the rest to our fellow coders on the forum.


Cheers


Mathias

I'm a very good housekeeper !
Each time I get a divorce, I keep the house

Zaza Gabor

Re: Import and Export

Hi Mathias,


Thank you very much...........................
I have tried few things to apply your code to import beside export but couldn't get it working.
Please see the attached sample project.

Post's attachments

Attachment icon Import-Export Dmitry - Work2.zip 36.27 kb, 458 downloads since 2018-01-28 

Adam
God... please help me become the person my dog thinks I am.

4 (edited by mathmathou 2018-01-29 00:02:02)

Re: Import and Export

Hello Adam,


Using StringList for exporting was only to get all data from X tables in a single List and exporting them to a single CSV file.


When importing, you need to differentiate the DB tables where the imported data will be saved, the StringList trick does not work anymore, at least not this way.


Furthermore, I see in your Form1_bImport_OnClick procedure a bunch of SELECT, while you are importing from a CSV file and saving to a database. Shouldn't those instructions be INSERT instead of SELECT ?


Importing is a whole different story. Using StringList would only bee useful if you want to compare to lists : the one to be imported, and the one built from database data to compare and find duplicates before saving.


Have a look at the IndexOf() function in TStrings : it lets you check for the presence of a certain string in another one. If indexOf =  - 1, the string was not found and then it's not a duplicate.


The general idea is this :
1 - build a stringlist with database data (this will give you all your fields separated by commas)
2 - build a second stringlist with data from the CSV file
3 - compare second list with first one to try and find sequence of this list in the original one
4 - if IndexOf = - 1 then you can save to DB because this is new data
5 - if IndexOf <> - 1 then this is duplicate data, discard it.


Try your hands on that, it's a very good exercise and you will need that often. if you encounter some problems, I'll try to find some time to build a simple example for you.


Furthermore, those lists have known counts, which could help you the progressbar you talked about in your first post.


Cheers


Mathias

I'm a very good housekeeper !
Each time I get a divorce, I keep the house

Zaza Gabor

Re: Import and Export

Hi Mathias,


Thank you for the detailed info but it seems I'm not there yet.


I was hoping to make it work with bit of less script when exporting multiple tables fields without the use of StringList  but I couldn't get it working in syntax front - I think?

Export('SELECT lastname, firstname, salary, dateofbirth, isSmoke, comment FROM employees')+#13#10+
       ('SELECT coDate, coName, coActive FROM Company', 'ExportedData-' + FormatDateTime('dd-mm-yyyy hh-nn-ss', now)+'.csv');

Anyhow, I tried to get import working but sadly failed in that front too. It returns error saying "...The system cannot find the file specified." after attempting to import exported csv file.

var SaveDialog: TSaveDialog;
    OpenDialog: TOpenDialog;
    sFileName: string;
    Results: TDataSet;
    i,c: integer;
    sValue, sValues: string;

Procedure ExportToSl (SQL: string; Sl: TstringList);
begin
SQLQuery(sql, Results);
c := Results.FieldCount-1;

  while not Results.eof do
  begin
  sValues := '';
  for i := 0 to c do
      begin
      sValue := Results.Fields.Fields[i].AsString;
      if (Results.Fields.Fields[i].DataType = ftFloat) or (Results.Fields.Fields[i].DataType = ftLargeint) then
          sValue := ReplaceStr(sValue, ',', '.')
          else
          begin
          sValue := ReplaceStr(sValue, ',', '&comma;');
          sValue := ReplaceStr(sValue, #13#10, '|');
          sValue := '''' + escape_special_characters(sValue) + '''';
          end;

          sValues := sValues + sValue+',';
        end;

  if sValues <> '' then SetLength(sValues, Length(sValues)-1);
  sl.Add(sValues);
  Results.next;
  end;
Results.Close;
end;


// EXPORT ///////////////////////////////////////////////////////////////////////////////////////////////////////////
procedure Form1_bExport_OnClick (Sender: string; var Cancel: boolean);
var Sl : TstringList;
begin
Sl := TstringList.Create;
sFileName := 'ExportedData-' + FormatDateTime('dd-mm-yyyy hh-nn-ss', now)+'.csv';
SaveDialog := TSaveDialog.Create(Form1);
SaveDialog.FileName := sFileName;
SaveDialog.DefaultExt := 'csv';
SaveDialog.Filter := 'Database|*.csv|Any file|*.*';
SaveDialog.Options := ofOverwritePrompt+ofHideReadOnly+ofEnableSizing;

if SaveDialog.Execute then
   begin
       ExportToSl('SELECT lastname, firstname, salary, dateofbirth, isSmoke, comment FROM employees', Sl);
       ExportToSl('SELECT coDate,  coName, coActive FROM Company', Sl);
       Sl.SavetoFile('.\ExportedData.csv' + FormatDateTime('dd-mm-yyyy hh-nn-ss', now)+'.csv');

       ShowMessage('Data has been exported in to the specified folder');
   end;
SaveDialog.Free;
Sl.Free;
end;


// IMPORT //////////////////////////////////////////////////////////////////////////////////////////////////////////
procedure Form1_bImport_OnClick (Sender: string; var Cancel: boolean);
begin
sFileName := 'ExportedData-' + FormatDateTime('dd-mm-yyyy hh-nn-ss', now)+'.csv';
OpenDialog := TOpenDialog.Create(Form1);
OpenDialog.FileName := sFileName;
OpenDialog.DefaultExt := 'csv';
OpenDialog.Filter := 'Database|*.csv|Any file|*.*';
OpenDialog.Options := ofOverwritePrompt+ofHideReadOnly+ofEnableSizing;

if OpenDialog.Execute then
   begin
   Import('employees', 'lastname,firstname,salary,dateofbirth,isSmoke,comment'+#13#10+
          'Company', 'coDate, coLogo, coName, coActive', 'ExportedData-' + FormatDateTime('dd-mm-yyyy hh-nn-ss', now)+'.csv');

   CopyFile(ExtractFilePath(Application.ExeName)+'ExportedData.csv' + FormatDateTime('dd-mm-yyyy hh-nn-ss', now)+'.csv', OpenDialog.FileName);
   ShowMessage('Data has been imported.');
   end;
OpenDialog.Free;
end; 
Adam
God... please help me become the person my dog thinks I am.

Re: Import and Export

Hello Adam,


You can not (I think) through 2 SQLExecute at the same time in the same function. If you really want to query multiple tables at the same time in one query, the best option is to use the UNION keyword between your queries. But this will only work if the tables have the same fields, which is not the case. I think you'll have to consider working with stringlists.


Then, when you call (on import)

sFileName := 'ExportedData-' + FormatDateTime('dd-mm-yyyy hh-nn-ss', now)+'.csv';

you refer to a file which name is composed of the date and time of your search, probably a few seconds after the export, so the systme can not find it because it does not exist. The export file was generated a few seconds earlier, thus the name is different. The problem here is the 'now' variable you are using.


A few lines down, you write :

CopyFile(ExtractFilePath(Application.ExeName)+'ExportedData.csv' + FormatDateTime('dd-mm-yyyy hh-nn-ss', now)+'.csv', OpenDialog.FileName);
   ShowMessage('Data has been imported.');

Wrong name again, because you added '.csv' twice : once just after Application.Exename and another time at the end.


Come on my friend, you know better than that, I'm sure smile


Anyway, I'll try to through a couple of lines for you tonight.


Cheers


Mathias

I'm a very good housekeeper !
Each time I get a divorce, I keep the house

Zaza Gabor

Re: Import and Export

Hi Mathias,


you refer to a file which name is composed of the date and time of your search, probably a few seconds after the export, so the systme can not find it because it does not exist. The export file was generated a few seconds earlier, thus the name is different. The problem here is the 'now' variable you are using.

I didn't realize that. Thanks...


I took out date and time from file name but still got same  "...The system cannot find the file specified." error tho

// EXPORT ///////////////////////////////////////////////////////////////////////////////////////////////////////////
procedure Form1_bExport_OnClick (Sender: string; var Cancel: boolean);
var Sl : TstringList;
begin
Sl := TstringList.Create;
sFileName := 'ExportedData.csv';
SaveDialog := TSaveDialog.Create(Form1);
SaveDialog.FileName := sFileName;
SaveDialog.DefaultExt := 'csv';
SaveDialog.Filter := 'Database|*.csv|Any file|*.*';
SaveDialog.Options := ofOverwritePrompt+ofHideReadOnly+ofEnableSizing;

if SaveDialog.Execute then
   begin
       ExportToSl('SELECT lastname, firstname, salary, dateofbirth, isSmoke, comment FROM employees', Sl);
       ExportToSl('SELECT coDate,  coName, coActive FROM Company', Sl);
       Sl.SavetoFile('.\ExportedData.csv');

       ShowMessage('Data has been exported in to the specified folder');
   end;
SaveDialog.Free;
Sl.Free;
end;
                      

// IMPORT //////////////////////////////////////////////////////////////////////////////////////////////////////////
procedure Form1_bImport_OnClick (Sender: string; var Cancel: boolean);
begin
sFileName := 'ExportedData.csv';
OpenDialog := TOpenDialog.Create(Form1);
OpenDialog.FileName := sFileName;
OpenDialog.DefaultExt := 'csv';
OpenDialog.Filter := 'Database|*.csv|Any file|*.*';
OpenDialog.Options := ofOverwritePrompt+ofHideReadOnly+ofEnableSizing;

if OpenDialog.Execute then
   begin
   Import('employees', 'lastname,firstname,salary,dateofbirth,isSmoke,comment'+#13#10+
          'Company', 'coDate, coLogo, coName, coActive');

   CopyFile(ExtractFilePath(Application.ExeName)+'ExportedData.csv', OpenDialog.FileName);
   ShowMessage('Data has been imported.');
   end;
OpenDialog.Free;
end;
Adam
God... please help me become the person my dog thinks I am.

Re: Import and Export

Hello Adam,


I have a project for you but it is nearly 1 in the morning and I have to clean and comment the code.
Tomorrow is a day of for me, I’ll clean up everything and post it for you first thing with morning coffee smile


Export, import with duplicate check via stringlists and progressbar.


Cheers


Math

I'm a very good housekeeper !
Each time I get a divorce, I keep the house

Zaza Gabor

Re: Import and Export

Hi Math,


I have a project for you but it is nearly 1 in the morning and I have to clean and comment the code.
Tomorrow is a day of for me, I’ll clean up everything and post it for you first thing with morning coffee smile


Export, import with duplicate check via stringlists and progressbar.


Thanks a lot...... Looking forward to it

Adam
God... please help me become the person my dog thinks I am.

Re: Import and Export

Adam,
I had a chance to look at your project. I'm curious as to what is the purpose for exporting the entire database into a single csv file.


I took a slightly different approach but still using some of the concepts which Mathias suggested. In the attached project I coded it in such a way that you do not need to manually put in SELECT or INSERT statements for all your tables. and fields. The script will automatically retrieve all tables and columns within each table and build a single csv file for exporting.


As far as for importing, I had to create header lines for each table in the export procedure in order to distinguish between tables when reading the single csv file during the import process.


I used  a different process for checking duplicates when importing. The import process will do an SQL SELECT to see if the record exists in the respective table. It checks all fields (except id) against the database table for equal values.


You should be able to transport the import-export scripts to any project. It's not dependent on specific table definitions. However, keep in mind that images really cannot be copied to csv files. Hope this helps.

Post's attachments

Attachment icon Import-Export All Tables.zip 613.08 kb, 522 downloads since 2018-02-01 

Re: Import and Export

Hello...
I do not know if it is my PC or if it is the example, but when I import the CSV it does not restore the images of the companies.

Re: Import and Export

Hi EHW,


Thank you very much for your kind help....................


At my end, when I try to import exported csv file I get an error message:
"Variant of safe array index out of bounds."


Hi Wenchester21,

I do not know if it is my PC or if it is the example, but when I import the CSV it does not restore the images of the companies.

As EHW pointed out, images cannot be exported in .csv file, as it's a text file.

Adam
God... please help me become the person my dog thinks I am.

Re: Import and Export

Adam,
Are you sure you are importing a csv file which was exported from the new app? It won't work with an old csv file. It seems like it must have worked for Wenchester21 with your sample db. If you want, upload your SQLite.db, if it's not too large, and I'll try it here.

Re: Import and Export

AD1408 wrote:

Hi Wenchester21,

I do not know if it is my PC or if it is the example, but when I import the CSV it does not restore the images of the companies.

As EHW pointed out, images cannot be exported in .csv file, as it's a text file.

Hello Adam...
Thank you for clarifying the doubt about the images. Ahhh, I did not get the error you said, in fact it exports me and matters well to the example of the ehwagner.

Hello EHW...
Now, I tried to adapt the example to my work and I showed an error, I explain.
When I export the data, then I delete them from the database, and I try to restore them (import), I get an error "Foreign key error", see the example I upload and do the same as me, do they get the same error?

Post's attachments

Attachment icon Import-Export All Tables (Foreign key error).rar 306.51 kb, 547 downloads since 2018-02-02 

Re: Import and Export

Wenchester21,
Yes, you're right. That's why I asked Adam what the purpose of exporting the entire database to a csv file. If it's for backup purposes, then the app needs to be changed to incorporate the id's in each table. Usually exporting to and importing from processes are built to interact with third party software and then it's typically only certain tables that are involved. I will update the app to include id's, which should take care of your foreign key error.

16 (edited by wenchester21 2018-02-02 20:34:36)

Re: Import and Export

EHW ...
Brother, when you have a time look at this, my last post
http://myvisualdatabase.com/forum/viewtopic.php?id=4024

17 (edited by AD1408 2018-02-03 03:05:22)

Re: Import and Export

Hi EHW,


My apologies, error was due to me trying to import wrong database.


I only tested it on sample project DB I have uploaded here.
Looking forward to your update you mentioned when replying to wenchester21.


I understand it's not an alternative to back up. As far as I know having data  in csv format would be useful. Especially, when data needs to be migrated to a different DB app which is using different DB system.


ps/. Is it possible to use win save dialog instead of the one you have used when exporting?

Adam
God... please help me become the person my dog thinks I am.

Re: Import and Export

Adam and wenchester21,
I updated the app to include id's. The question becomes what constitutes a duplicate. Obviously if an id exists in the table then it is a duplicate. This would be the case if you are using the export out of this app and importing back in. However, I have allowed for records being created in the csv file outside of MVD, in which case the id would be blank. With this scenario the app checks for duplicates for all fields in the csv record against the table record. I have not done extensive testing. Just don't have the time right now. If you find something broken, let me know, but it may be a couple of days till I can get to it. Of course, if anybody else wants to step in, I'm ok with that.

Post's attachments

Attachment icon Import-Export_Revised.zip 369.94 kb, 588 downloads since 2018-02-03 

Re: Import and Export

I'm apologizing for bad English.

Dear ehwagner. I used your example for my project, but there was such a question. When importing, if the field in the (.csv) file is empty (, '',) then it is not marked in the database in any way, but it needs to be set to "NULL". Is it possible to complete this script? Otherwise, this leads to an error when duplicating records in the TableGrid.