Multiple instances of app and one db across i7's processors

Hi all,
I'm trying to run multiple instances of a single (statistical) application on an Intel i7 machine so the instances can take advantages of all the processors. I have one wrinkle where I hope all the experts here would be able to help out. This application uses an MS Access db where all the data is stored. Ideally, it would be great if Vista and i7 would parallel process one instance of the app and the single db across all the processors. But after reading what multiple processors really means for today's apps, I realize that multiple processing/threading is an application issue. This is a statistical application that's not programmed for multi-processnig/threading.

So far, I was able to run two instances of the app against the single db. One instance as the admin user (I'm not sure if my terminology is correct) and another as admin. This is within the same user account. Both instances run different statistics against the same db on different processors. So i'm maximizing two processors.

Now can the experts out there tell me how I could launch two more instances of that app while still using the same db? I tried launching a third within the admin user accout but it said one instance already running. So I tried launching the third as admin but again said instance running. I also created another user account and ran it but it couldn't open the existing db. It need another db. I tried running as admin and still same. Any ideas?

Also, the i7 has 8 virtual processors which are 4 physical processors so does that mean I can actually run 8 instances to maximize the 8 processors or do i have a limit of 4?

Regards.

Interesting question.

First, I'd suggest being absolutely sure whether the app itself is multi-threaded. Perfmon can help with that. If you watch the "number of threads" counter for the app's process instance (under the PROCESS perfmon object), it will give you a simple count.

If it's anything more than one single thread, the next question is whether more than one thread does computational work when the app is under load. In other words, is more than one thread doing number crunching or is only a single thread chewing CPU while another 20 sit around and do not-very-much at the same time. That's a function of the app's design. The THREAD object in perfmon will allow you to zoom in on each individual thread's processor utilisation. Ideally, when under load, you'll see multiple threads consuming substantial amounts of processor time - 20% or above.

Otherwise, if it's a single-threaded app, or if it's limited to doing heavy computational work on one thread, then I think you're probably out of luck. Database locking and contention are the limiting factors. The app may or may not have been designed to have multiple DBMS processes working concurrently with the same database. If it's something on a relatively small scale, odds are it hasn't been designed to allow for the necessary locking and resource arbitration.

Large SQL servers and distributed enterprise apps are designed to scale with available hardware. That's part of their appeal ($$$). Custom apps for small environments are rarely designed to be able to scale out to massive hardware capacities.

Hi H2SO4,
Thanks for your reply. You've just taught a man to fish with the thread info. It was tough to find how to do it but I figured it out.

The app is multi-threaded for calculations. But not for an optimization piece which is extremely time intensive. It's neural network learning. The optmization happens on a single thread, just like you said. It's a bottleneck.

As for db contention and locking, there doesn't seem to be. I'll explain. I'm able to submit a list of data sets to optimize. The app queues them up and optimizes them one at a time using the one thread. Once optimization is complete other calcs are done using multiple threads so very fast.

What's peculiar (but good) is that I can launch the app a second time as admin. This instance uses the same db and it continues to process the next data set in the queue in parallel to the first instance. So my machine is using 2of4 (4of8?) cores.

I wish there was a way to get other instances to run just like the second one. Then the others could have their own worker thread and I'd maximize my i7. But it just won't. I just tried the Allowdup command but no luck.

Any idea on how two instances are running ... one as admin user and another as admin? I guess if someone can tell me that then perhaps it could lead to being able to create other instances.

Now I can create a whole new db and get multiple instances but that's very cumbersome due because I can't compare results across the result sets.

Can I create another admin account within this same root user (terminology?) account where i've launched two instances already?

Any thoughts?

sajosh said:

Hi H2SO4,
...
Any idea on how two instances are running ... one as admin user and another as admin? I guess if someone can tell me that then perhaps it could lead to being able to create other instances.

Click to expand...

Let me turn the question around on you in a way which might make the situation clearer: how is this app able to limit its number of instances? Obviously, it checks how many ProcessX instances are active, and once some criteria are reached it disallows any more instances of itself to be started.

Notepad, as an example, does not implement any such throttling. You can start as many instances of Notepad as you like, and each one will show up as a separate process. Most apps act this way – they simply don’t care whether other instances are already running. Then there’s the “idempotent” processes like Explorer.exe. It’s almost impossible to start a second instance of Explorer because it has a mechanism for ensuring only one is instance is running at any time…

The way that’s usually done is to implement a “mutex” synchronisation object. Being MUTually EXclusive means a mutex can only be owned by one process at a time (there’s a lot more complexity but it’s irrelevant). When the first instance of ProcessX starts, it creates a named mutex, say “ProcessX_mutex” and it assumes ownership of that mutex. During its own attempted initialisation, a second ProcessX instance will also try to create the same mutex and grab ownership, but that will fail because the mutex is already owned by the first instance. The second instance will simply terminate, silently, without the user ever realizing that it attempted to start. Similar principles can be extended to any arbitrary number of maximum instances, or to any app-specific limitation such as “no more than two processes per database”.

Your app apparently implements this type of protection to prevent more than two processes from concurrently accessing the same DB. It may be a safety thing (locking is only implemented in a two process context), or it may even be licensing (the version which scales to more processors costs more). Either way, you’d have to ask the app vendor exactly how the throttling mechanism is implemented, and whether it is feasible and safe to disable it so you can have more instances working against the same database.

Remember that without adequate synchronisation it is entirely possible for two (or more) threads to mess up a data structure they are both accessing. That can even happen on a single-processor machine because the first thread might get pre-empted half-way through reading a customer record, only to have the second thread come along and alter the entire record before the first thread resumes execution, thereby corrupting the first thread’s view of the data.

sajosh said:

Can I create another admin account within this same root user (terminology?) account where i've launched two instances already?

Any thoughts?

Click to expand...

To be frank, I suspect that the fact that a second instance (in an admin context) actually works may be a bug in the app. They may be creating a mutex (or other synchronisation object) in a session-specific namespace, and they never intended for the concurrent-access-as-admin to work. In that case, you may be running the risk of logical DB corruption even with just two instances, let alone more.

Asking the app vendor is the best approach here. Good luck with it

What you're saying makes sense.

I've called the vendor who doesn't say much but simply advises to create another db. I can but the result sets will be fragmented and the workload for me goes way up.

I agree about your safety and locking db logic. THey use MS Access (circa 2000) The chance is there but the two instances seem to play nice with eachother. The data sets don't seem to get corrupt, empirically speaking.

I've 'hacked' two instances to play nice with eachother. So i'm looking to hack a couple more. The vendor thinks it's a windows question.

Is there a way to get more 'concurrent-access-as-admin' instances?

To clarify, I'm asking if there's a way to get more 'concurrent-access-as-admin' instances just so I can test it out and see empirically if there's a negative effect.

It depends on what's doing the limiting.

If it's the app itself, you'd literally have to find the specific mechanism and to circumvent it, but now we're in "potential breach of contract" territory. Having said that, WinObj is a relatively simple utility which would allow you to inspect objects created within the "Object Manager" namespace, to check whether there's a named mutex or some similar sync object created when an instance of the app runs:

WinObj

Killing the mutex would require injecting code into the process, or even latching on with a debugger to step through and prevent the mutex being created in the first place every time the app starts. Naturally, none of this is practical, especially if you haven't worked on that level before, but if the app is "protecting" itself I can't see an easier method.

On the other hand, it's curious that the vendor thinks this is a Windows limitation. Obviously they've never tried launching multiple instances of Notepad

I wonder whether it comes down to some sort of shared/exclusive access that is a side-effect of the flags they're passing in their CreateFile() calls. In other words, the first instance calls for exclusive file access, but the second manages to override because it's in the admin context...I dunno.

Here's a test: what happens if you launch the admin instance first - are you still able to start the non-admin instance afterwards and have it work with the same DB?

Good test. Couldn't run the second instance after the first started as admin. So it's the admin that can override the regular user. so is there a way to create multiple admin accounts?

Yeah, the mutex and injecting code stuff might be a bit much but i'm somewhat technical so with some help could figure it out.

I think what the vendor ment was that the 'hack' or bug exploit is happening at the windows level. Which we see is the case with the current test.

What next?

Oh I should mention that I can have multiple instances of the app running within the same user account. It'll just ask me to open another db. If I don't then it closes.

Given the description as it stands now, I'm relatively confident that it's only a bug in the app which is allowing a second instance (as admin) to access the same DB as the first (non-admin user) instance. Otherwise, there's little sense in allowing up-to-two concurrent sessions, but only if the non-admin session starts first.

Whether that's because of where they're creating the mutex in the wrong place, or indeed whether it's perhaps got nothing to do with mutexes but is instead related to dwShareMode settings in their CreateFile() calls, or similar, I can't tell from here. Perhaps the app is capable of some form of logging which would tell you why it exits on startup when the 3rd session is attempted, or you could even run it under a debugger if you have experience with that type of analysis.

My suspicion is that even with only the two instances doing concurrent access, it's only a matter of time before the DB is logically corrupted because of a lack of in-built synchronisation and concurrency mechanisms. There may be no errors at all - just inconsistent data somewhere in the DB.

Should you manage to work out how to attach 3 (or 4, 5, 6...) instances to the same DB, the likelihood of corruption will simply increase because of more contention. Data integrity is presumably a prime criterion for you, which is why I'd advise you not to go in this direction.

One thing I know for certain is that this is not a Windows limitation. It's their code and it doesn't exactly instill confidence when the vendor doesn't seem to realise that. Developers like that are unlikely to have "accidentally" added synchronisation code to protect against mangling by concurrent multi-process access, just-in-case someone works out how to launch multiple instances of their app with the same DB

You're absolutely right. Data integrity is top priority. It does make sense that sooner or later something within the data set could get corrupt. So I'll just live with multiple dbs.

If you happen to come across a way to add another admin instance, it would be fun to just test this out, more from curiosity than implementation.

Thanks very much for the thought you put into this investigation.
Regards.

Once again you make an crystal clear points on db corruptions. You're correct that data integrity is priority. So I took your lead and started to think about how the data might be stored and it seems (i'm just guessing here) that the data should be partitioned pretty nicely in the db. Here's why.

The more I thougth about your toughts I realize that I've really got a few dozen seperate data sets. They have a hierarchy of time period, type and study. I also tested this out. It seems that if I run the two instances then tell each to run statistics on different data sets (instance 1 runs all time periods 5 and instance 2 runs all time periods 15 where both are distinct data sets) then their doesn't seem to be any logical contention for records. So instance 1 works in it's own section of the db and so does 2.

As long as I don't ask both instances to run on the same data set, it would seem logical (to me, a non expert) that they'll play nice with eachother on the db. Same goes with instances 3 and 4. What do you think?

Tests on 2 instances seem to go well so I'd like to try 3 or 4. Would you know how to launch multiple instances as other concurrent admins?

Have some rep, to celebrate the fact that my "rep power" has hit double digits - I admire your dedication and the way you're thinking about the mechanics of the issue

OK, let's leave the DB integrity question aside for now, under the assumption that no two processes would have reason to access the same portion of the DB, like you said. I still think there'd be metadata structures which would be in danger of corruption, but I don't know enough about Access DB internals to provide any useful input there.

Regarding getting past the two instance limit, it would be necessary to analyse the app in order to find the limiting mechanism first. Since the vendor did not mention any licensing limitations in this regard (they think it's a Windows issue), presumably you're in the clear from a legal standpoint if you do manage to add more instances. Also, it's perfectly legal to run code on your own machine under a debugger, and that's what you'd have to do to find why the 3rd instance fails to launch successfully.

The easiest option would be to crank up the logging level on the app, if it's got any logging that is. You may find that the 3rd (failed) instance produces a log file stub which might contain useful info such as (entirely hypothetically):

<time > Calling CreateFile() to gain access to c:\MyDB\DB.mdb
<time+1> ERROR: CreateFile() returned "File is locked for exclusive use"
<time+2> ERROR: Unable to continue initialisation. Terminating...

Which would confirm that the way the app is opening the DB file from the other two instances is preventing the 3rd, and the fact the admin instance works is actually a bug in the app (hehe).

If no such logging facility exists in the app, then you're really limited to analysing the startup of the 3rd instance under a debugger. It's not rocket science, but unless you've debugged Win32 software before the learning curve is going to be near vertical, and it's just not practical to help you through a web forum. Having said that, I suspect you're in a university environment and no doubt surrounded by smart people. Some of them will know how to run an app under a debugger and set the right breakpoints.

Then there's always trial-and-error, but I dislike that personally, and I can't think of too many obvious things you could try. Removing all anti-virus and anti-malware utilities as a temporary test would be useful, in case they're responsible for the file locking in the first place (it's possible). You should also try launching the 3rd instance in a separate admin account context (type RUNAS on a CMD prompt to see how to do that).