[daisy] issue with large amout of documents

[Ross Singer]

On 6/7/07, Bruno Dumon <bruno at outerthought.org> wrote:

> >   Perhaps if i go directly to the database in stead of working via the
> > api?
>
> Then there's not much point in using the repository server: just use an
> RDBMS then.
>

This is a strange response.  Could you not just get back the document
IDs from the RDBMS and then go back to interfacing with the repository
server?  I'm not sure how that would be in opposition to 'using the
repository server'.

I guess the problem I see is that Daisy doesn't handle large document
result sets sanely at all, so there's little recourse /but/ to find
some sort of alternative to the API when dealing with them.

For example, in our setup, we have about 750K documents (of what would
be around 2M total) that we might need to extract and either delete
and resubmit or possibly tweak.  Since LIMITs don't apply until
/after/ the query has taken place, it's nearly impossible to
effectively handle this.

What do you suggest doing in these scenarios?

-Ross.