[menonshyam]

Hello all,

I am trying to use sql database files (.db) with OpenFOAM for my application. I am able to run codes that successfully read from the database files during the simulation. It also works with parallel processors. However, I suspect that the simulation slows down since all processors are accessing the database file from the same location (in my case folder). Is there perhaps a way to distribute the database file in each processor folder so as to speed up operation. I tried copying the database file to each processor folder but it is not being read. I use the following code in "main" to open the database file:

	if (sqlite3_open("Database.db", &db) != SQLITE_OK) 
	{
		std::cerr << "Could not open database.\n";
		return 1;
	}

Is there a way to specify a path in the sqlite3_open statement that could point to the database file in each specific processor folder?

Thanks!
Shyam

[jhoepken]

Hi Shyam,

I've had a somewhat similar problem and switched from a SQLite database to a mySQL database, for that very reason. This database is not hosted on the machines I use to run the simulations on, which distributes the IO.

Further on I created a data structure that takes care of the data being only read once *and* only by the master process (Pstream::master()). I then use Pstream::scatterList, to scatter the data, which is stored in a List<T>, to all processes.

Maybe this helps.

Jens

[menonshyam]

Hello Jens,

Thank you for your reply.

It seems that mySQL is installed and available on the processors i use to run my simulations. I'll have to see how much of the SQL programming code i need to change to go from SQLite to mySQL!

However, in my code i go through cell and patch arrays and query the table in a sequential manner every time step. Since the queries themselves change every time step, I do not think i can query just once like in your simulation. Else, I would have to carry around a lot of data which is something i want to avoid.

I am still hoping that there is a way to allow each processor to access their own copy of the database. There appears to be a distributed data option in OF. Not sure if that can accomplish what i need.

Thanks!
Shyam

[jhoepken]

Hi,

code change: I would strongly suggest, that you create a class hierarchy, that wraps the database communication in a separate layer. Doing so enables you to change the type of database later on, without affecting the rest of the code.

mySQL vs SQLite: I doubt that you would have to change a lot of the SQL syntax.

Querying: I do the same stuff, basically. That's why I cache the data on the master node. Otherwise the database communication - in the case of my application - way to dominant. The entire process would be slow as hell.

Access: If you can really separate the data on down to each processor, you should be fairly safe. mySQL can handle quite a lot of queries.