Forums » Ferret » Substantial problems with write locking (and other flux)

I am having some great trouble keeping my Ferret indexer for 
ActiveRecord working.

First the get_field_names disappears (now back), then I am collectig 
some major trouble with locking. Same thing here:

exception 6 not handled: Could not obtain write lock when trying to 
write index

A snippet like this just deadlocks retrying endlessly:

  begin
    @ferret_index << doc
    @ferret_index.flush()
    @ferret_index.close()
  rescue Exception => e # No, he couldn't define a proper class for this
    if e.to_s.include?('Could not obtain write lock')
       reopen_index # opens the index again!
       retry
    else
      raise e
    end
  end

How are we supposed to handle concurrency with a file store? I can't 
find anything in the wiki and actually I am getting very frustrated. 
It's the third gem update of Ferret and my plugin just got broken, I 
can't repair it since.

Also the habit of throwing Exceptions is somewhat obnoxious because they 
are not standard errors. Might Ferret once get his own error class tree?

On 6/30/06, Julik <me@julik.nl> wrote:
> I am having some great trouble keeping my Ferret indexer for
> ActiveRecord working.
>
> First the get_field_names disappears (now back), then I am collectig
> some major trouble with locking. Same thing here:

I ported 20,000 lines of ruby code to C. I apologize if some parts of
the API got left out in the process. People wanted a faster search
library so that's what I'm working on. It's still beta, although it
should still be alpha.

> exception 6 not handled: Could not obtain write lock when trying to
> write index

Do you have more than one process writing to the index? The C version
of Ferret currently doesn't wait long enough for the lock to be
released. This behaviour will improve once I get the current release
finished although I'm guessing that locking problems are always going
to occur.

>     else
>       raise e
>     end
>   end



> How are we supposed to handle concurrency with a file store? I can't
> find anything in the wiki and actually I am getting very frustrated.
> It's the third gem update of Ferret and my plugin just got broken, I
> can't repair it since.
>
> Also the habit of throwing Exceptions is somewhat obnoxious because they
> are not standard errors. Might Ferret once get his own error class tree?

I'm working on this too. I've done a significant rewrite of cFerret so
that it will gel better with Ruby. I've basically rewritten 10,000 LOC
which is why I haven't had time to respond to the most recent tickets.
All the problems people are having are going to be addressed.
Exception handling will be significantly improved. Hopefully I'll be
able to eliminate segfaults. A windows version is also on the way. So
please be patient. Or fix it yourself and send in a patch.

Cheers,
Dave

David Balmain wrote:
> It's still beta, although it
> should still be alpha.

Got the message :-)

> Do you have more than one process writing to the index? The C version
> of Ferret currently doesn't wait long enough for the lock to be
> released. This behaviour will improve once I get the current release
> finished although I'm guessing that locking problems are always going
> to occur.

No, it throws within a unit test which runs inside a single process. I 
ain't even got to any real concurrency yet :-)

> I'm working on this too. I've done a significant rewrite of cFerret so
> that it will gel better with Ruby. I've basically rewritten 10,000 LOC
> which is why I haven't had time to respond to the most recent tickets.

I realise I vented too much, so apologies for that. Great kudos to you 
for the work you are doing on Ferret.

Would you please put out a gem with the "get_field_names" back in the 
picture? This is the only thing broken after some major patching I did 
on my plugin.

Would you be so kind to give me a tip what "get_field_names" ought to 
return when casted to an Array so that I can replicate it's 
functionality (I have the names of fields stored elsewhere, I just need 
to pass them when constructing the search).

TIA.

Julik wrote:

> No, it throws within a unit test which runs inside a single process. I 
> ain't even got to any real concurrency yet :-)

And to be fair - the problem seems to be removed altogether by using 
auto_flush. So now the get_field_names thing is the only one and then I 
can hack further and switch an app to Ferret.

On 7/1/06, Julik <me@julik.nl> wrote:
> Julik wrote:
>
> > No, it throws within a unit test which runs inside a single process. I
> > ain't even got to any real concurrency yet :-)
>
> And to be fair - the problem seems to be removed altogether by using
> auto_flush. So now the get_field_names thing is the only one and then I
> can hack further and switch an app to Ferret.

I've released the Gem. I'm not sure how much has changed since the
last release as it's been so long since I did any work on that branch,
but it does have the get_field_names method and it raises a
StandardError instead of Exception. But if I were you, I'd wait around
for the next release of Ferret and possibly even join in the
discussions on the API. There is going to be 0.10.x string of releases
which will break backwards compatibility with 0.9.x and earlier. The
goal is to get out a 1.0 release with a stable API. I'll then come
back and fix the bugs in 0.9.x for people who aren't able to change
over, but I won't be supporting it for very long.

Cheers,
Dave