[Fwd: Re: [translate-pootle] Wordforge]

F Wolff · External Since: May 26, 2006 Posts: 3

I just saw that I forgot to CC the debian list with my reply of
yesterday.

On Do, 2006-06-08 at 00:17 +0300, Gintautas Miliauskas wrote:
> Hi,
>

Hi

There are a lot of things to reply to. I've CC'd the developers list
(this is the users list).

Hopefully the other people will respond as well to give a more thorough
reply.

> > As the others have mentioned, yes, we are doing 2 major restructure:
> >
> > 1) base classing of our convertor classes and Pootle
> > 2) Locking
> >
> > Some of this does affect backend separation.
> >
> > There of course is lots of work, but I understand that yours needs to
> > be quite independent. We have been discussing the concept of change
> > queues which relates to locking ie we don't want to step on someone
> > else's change if for some reason you had yours out for too long.
> > Queues would also allow us to create a distributed environment. This
> > direction could be the most fruitful for your work.
> >
> > But as the others have said #pootle is a good place to ask.
>
> I have a strong opinion about the direction Pootle's backend should be
> headed. I think that at the moment you have a 'loose' system based on
> files, which is simple and transparent. However, it is inefficient
> in memory usage, speed and ease of distribution. Since memory usage
> depends on the database size, I take it that you are using some sort of
> memory cache to speed up the system.
>

Indeed. There is a cache.

> I think that an obvious solution here is to use a relational database
> (I would suggest PostgreSQL). Unlike ordinary files, it allows
> extremely speedy random writes which is exactly what we need here.
> I would expect the problem of high memory usage to disappear completely
> too. In fact, if we can put all important data on the database,
> distribution of the system would then become trivial -- several
> instances of the application (possibly on different computers) would
> simply use a single instance of the database. I would say that by doing
> everything (indexing, locking, etc.) manually we're reinventing the
> wheel, badly.
>

Well, we must remember that Pootle does a lot of things. I mostly agree
with what you are saying. We all understand why databases are good at
these things. But Pootle is also a file server. It doesn't necessarily
make sense to convert everything to a database and back each time you
want to do a file operation. For some people the file management is the
desirable part. We try to minimise diffs for the benefit of people
needing to review patches for upstream projects and things like that.
Some projects are _very_ picky about these things. So the files do have
a place I believe for some of our users. Of course we'll run into
problems when we have a thousand translators busy with interactive
translation, and we have already realised and discussed many of the
things you mention. Eventually we want to make Pootle as distributed as
possible, not just in the sense that you mean, but even with more
independent installations being able to interact.

As for specific technology - jToolkit (that we already use) has support
for many databases so we'll probably use that functionality so that we
don't become tied to any specific one.

> I also think that using XLIFF, an XML format, for the backend is a bad
> idea. I think that XML is great for serializing data and sharing it
> between completely disparate systems, but it's awful for random writes
> and places where performance is important (such as the data storage
> backend for a heavily used system). Nobody cares whether the backend
> storage is compliant with some standard, it's only the interface where
> standards-compliance matters. The backend must simply be as efficient
> as possible and not get in the way.
>

Here I agree. Some people I know would say XML isn't good for
anything Smile

I believe that eventually XLIFF would be used for
interchange only. Probably the same for TMX as well.

> I do not mean to thrash your design decisions or stall work on the
> backend. Files as backend are great for small projects where
> performance is unimportant, because then you don't need to set up an
> SQL server. I just want to suggest designing the API in such a way that
> does not depend on files, i.e., such that a relational database would
> not be too much trouble to plug in.
>

Yes. That is exactly what we have already started with to some extent.
Initially Pootle was entirely PO file based and of course one needs to
start small. All our tools worked with po files, so this was logical.
But now we have developed a more general API so that all our tools (not
just Pootle) can interact with them in a general way. Now hopefully a
database can become another storage mechanism that plugs into our API.
That way we also don't force people to start using a database if they
don't want/need to.

> I am concerned with this issue because my SoC project is for Debian,
> not for Pootle directly, so I will need a backend that would handle
> *all* Debian translations at the same time. That would mean gigabytes
> of data -- a "small" relational database by modern standards, but
> seemingly infeasible with the current structure because of huge
> resource usage. And I need it by the end of summer Wink

Obviously I will
> be working on this, but I would like to make sure in advance that you
> will not be working in the opposite direction.
>

No, I don't see our goals to be conflicting at all. Our previous public
installation of Pootle was getting slow later on. Recently I learnt that
it was running on a machine from the previous century Smile

(Pentium or
something). But of course the Debian files would mean a new scale of
operation. And I don't believe working on that is in conflict with what
we are doing. But I believe it is also useful to remain the current
approach as well, since I think it is exactly what some people want.

> During the discussion with Aigars in Riga he seemed to agree with the
> idea of hooking Pootle to a relational database.
>
> I would be happy to hear your thoughts. I hope the letter did not come
> out too harsh. There may be more options here, or you may have some
> plans that I am simply not aware of. However, this is critical for my
> work for Debian and I want to cover this ASAP.
>
> Best regards,

Thank you for your thorough posting. I'm not offended at all, if that is
what you feared Smile

Hopefully the others will also reply more
thoroughly. Just wanted to start the discussion.

F

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Otavio Salvador · External Since: Dec 01, 2004 Posts: 587

Gintautas Miliauskas <gintas.RemoveThis@akl.lt> writes:

I won't quote your message since I'll basically do a braindump like
you did previously.

I fully agree with you that we should avoid to store data directly on
files also because if we later discover another format will be very
difficult to reuse old data and we'll need to make converters for
it. So, let does it from basic design.

I like the idea to RDBMS is the way to store the data and the "final"
format (po or whatever) should be build just as a product of it. Also,
I agree about the idea to have a "server layer" between the client and
backend because it makes trivial for user to use the backend without
much hassle and without the need to control locks and whatever of
internal details of it.

One thing that come to my mind just now is that we might have the
following design:

|--------------------------------------|
| | Pootle itself |
| Clients |----------------------------|
| | Pootle compatibility layer |
|--------------------------------------|
| XML-RPC server | Other servers |
|--------------------------------------|
| API to access the data |
|--------------------------------------|
| RDMS with data |
|--------------------------------------|

With that in mind I realise that might be easier to get colaboration
of Pootle people since they can help us to design the layers bellow of
their compatibility layer and at same time improve their system in a
easier way without the needed careful to don't brake our system. So,
with that we both are basically free and able to help each other.

The only drawback on it that I can see is that we won't contribute
much code back to Pootle directly. Basically fixes to get it more
"backend agnostic" and then plug our backend there. That, IMHO, is a
good conseguence since they'll be free to choose and develop other
backends without the compromise to us just one.

What's your point of view about this?

--
O T A V I O S A L V A D O R
---------------------------------------------
E-mail: otavio.RemoveThis@debian.org UIN: 5906116
GNU/Linux User: 239058 GPG ID: 49A5F855
Home Page: http://www.freedom.ind.br/otavio
---------------------------------------------
"Microsoft gives you Windows ... Linux gives
you the whole house."

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Javier SOLA · External Since: Jun 10, 2006 Posts: 5

Hi Otavio, Guintautas,

I would like to propose a course of action.

1.- Gintautas works on the API (which has to include the separation of
front-end and back-end), and on the XML-RPC. This will really clear up
things for further work, and will facilitate the specification of a DB.
This work is done in a separate branch, as required by the project.
2.- The WordForge present staff will integrate these changes in Head as
soon as they are clear, so that Gintautas' branch and Head stay as close
as possible, and integration is not a problem later.
3.- Independently of the work that is continuing on WordForge (front-end
or file-based back-end), WordForge will be happy to fund somebody to
work on designing and implementing a RDB back-end. This can be somebody
working on parallel during the coming months, or Guintautas after he
finishes (if he has time before going back to school). Then we just use
-or continue working on-the best back-end that we have. We would work
together with this person to ensure that all the data that we need in
the project is present in the DB (even if most of should be defined
already defined by the API).

I somehow have a feeling that the optimal result will be something
between file-based and DB based, again, I could be completelly wrong. We
will not really know what it is until we try.

Some more comments below.

Otavio Salvador wrote:

>Gintautas Miliauskas <gintas.TakeThisOut@akl.lt> writes:
>I fully agree with you that we should avoid to store data directly on
>files also because if we later discover another format will be very
>difficult to reuse old data and we'll need to make converters for
>it. So, let does it from basic design.
>
>
I have to disagree with this. Independently of if the best storage is
files or DB, the standard formats are precisely done so that they will
last for a long time. XLIFF is an OASIS standard that will evolve, and
will not be replaced in the coming decades. Same as Unicode, the
standards are there to last... and, even if change, happened, you would
still need to write converters. On the contrary, when these standards
advance (within backwards compatibility), you will have to modify the
database to admin new types of information, but you would not need to
modify a file-based system

>I agree about the idea to have a "server layer" between the client and
>backend because it makes trivial for user to use the backend without
>much hassle and without the need to control locks and whatever of
>internal details of it.
>
>
Makes a lot of sense.

>One thing that come to my mind just now is that we might have the
>following design:
>
> |--------------------------------------|
> | | Pootle itself |
> | Clients |----------------------------|
> | | Pootle compatibility layer |
> |--------------------------------------|
> | XML-RPC server | Other servers |
> |--------------------------------------|
> | API to access the data |
> |--------------------------------------|
> | RDMS with data |
> |--------------------------------------|
>
>
>
Adding a little more meat:... Also, I would see the a separation between
the Pootle web-based file server and the Pootle editor, and maybe
putting them on top of the API, even if maybe this is not the best place
for the editor. Would this break the idea that you expressed with the
server layer above?

|-----------------------------------------------------------------------------|
| Clients | | CVS/SVN |
| Other Pootle | | FOSS projects |
| servers | | |
| Rosetta | Translators Reviewers, etc. | |
| Off-line Editors| | |
| Compiler farm | | |
|-----------------------------------------------------------------------------|
| XML-RPC server | Pootle file Server| Pootle Editor| Other servers + filters|
|-----------------------------------------------------------------------------|
| API to access the data |
|-----------------------------------------------------------------------------|
| RDMS with data | File based System | Other back-ends? |
|-----------------------------------------------------------------------------|

>With that in mind I realise that might be easier to get colaboration
>of Pootle people since they can help us to design the layers bellow of
>their compatibility layer and at same time improve their system in a
>easier way without the needed careful to don't brake our system. So,
>with that we both are basically free and able to help each other.
>
>
I agree, you can count on all our help.

>The only drawback on it that I can see is that we won't contribute
>much code back to Pootle directly. Basically fixes to get it more
>"backend agnostic" and then plug our backend there. That, IMHO, is a
>good conseguence since they'll be free to choose and develop other
>backends without the compromise to us just one.
>
>
Our idea -as I mentioned- is still that we work together in the
different options. We would not like to see an "our back- end" and a
"your back-end". The advantage of working together is that there are
more people thinking, and the good news is that we have sufficient
funding to look at both options, and then pick up the one that we all
agree is the best, and continue working in that direction.

Javier

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

Javier SOLA · External Since: Jun 10, 2006 Posts: 5

Hi Gintautas

Gintautas Miliauskas wrote:

>I do not really care about the actual implementation of the backend.
>The reason I am advocating an RDBMS is to influence the design of the
>backend. In the current situation, an API that just presents parsed
>files and writes objects back would do, but it would not be possible to
>have an efficient backend based on a database this way.
>
>
Independently of the backend storage, FOSS projects and translators (the
users and customers of the system) work with files, which must interact
with Gettext, translation editors, etc. If the API does not produce
files, then another layer on top will have to do it, creating a new
unnecessary layer. Same for the XML-RPC. If we want to exchange data
between servers, this data will relate to specific projects, and the
data will have to be grouped in the files that the project produces.

Only in the case of the on-line editor, strings might be addressed
directly, without having to construct the file, but contextual
information will stll be necessary in many cases. For example, when
translating a segmented text (such as a help page), it is important to
know the preceeding sentences and the ones that follow.

>Here's a single simple functional requirement that I propose for the
>backend:
>
>=============
>It should be possible to write a separate program which should be able
>to do everything possible only using the backend object. No
>assumptions on files, no nothing, just manipulating objects. I would
>not have to care about locking, other servers running in parallel
>using the database, the OS, the filesystem or anything else. That also
>includes uploads of new languages / new projects.
>=============
>
>Specifically, say I want an XML-RPC server. I just write a Python
>script that imports some classes from pootle.backend (or whatever),
>defines some functions and makes them public.
>
>It's a pretty basic requirement for any component: independence. Does
>your new design offer that?
>
>
Independence of... ?

>There is also a more technical question of the nature of the backend.
>It can be procedural (you call functions which return things), or it
>can be object-oriented: there are a few functions that return "basic"
>objects defined by interfaces rather than by implementation. Then
>you operate on those to get subobjects, to save the changes to the
>objects, etc. Which approach are you using?
>
>A few more notes:
>
>Trying to stick to a standard format (XLIFF, .po or anything else)
>for backend storage is not a good idea because it will be unnecessarily
>limiting. The standards have their own specifics which we may not
>care about, so there's the overhead of storing things the 'standard'
>way, even if it is not convenient. Eventually the format will simply
>not be enough. We will want to keep lots more metadata (e.g.,
>string history, string submitter, date, ...). Storing that in external
>files, separated from the actual data, will be increasingly
>uncomfortable.
>
>
You should read the standards. They are made by people who have been
working in localisation for many years, have extremelly clear
understanding of what is necessary, how to structure it and how it
should be encoded. We DO care about them. If there was no PO standard,
each FOSS application would have to do its own translation editor, and
we would not be here now. We produce standard files so that standard
translation editors can be developed and used. There is no modern
computer science without standards.

Standards are in constant review and evolution, making sure that new
types of data that might be necessary are implemented. There is not such
a thing as "eventually the format will not be enough". Beside standard
extensions, XLIFF and many other XML formats (such as say OpenDocument)
allow user extensions, for the cases in which the people who define the
format might have left anything out. We are working with XLIFF 1.1, but
XLIFF 2.0 is being worked on, even if very few changes will take place,
and all of them backwards compaltible.

The debate of files/DB backend is -nevertheless- independent from the
use of standards.

>Anyway, from the engineering point of view, there is 'primary' data,
>and there are 'views' on that data. Do not confuse the two!
>XLIFF, .po or .html are just views. Data is just that, data, it has no
>connection to a format until you serialize it. (An RDB is attractive
>as you don't have to serialize your data and commit to a format.).
>
>
You DO commit to a format, it just happens to have very efficient ways
of handling data (in general)

>Frankly I am extremely puzzled with what I perceive to be a hostile
>look towards RDBMSes. Some seem to be willing to jump through many
>hoops to defend file-based approaches. In fact I found conflicting
>advice in the wiki itself:
>
>
Please do not confuse being careful with being against something, and
please do not use words as "hostile". Some of us have been working on
localisation for quite a number of years, as well as in development,
databases and development of standards. We understand the complete set
of data that needs to be managed, something in which some of us have
been working for quite a while. The DB vs. files approach has been
discussed innumerable times, we are quite aware of the advantages of
databases.. and of their problems. Any argument that you might put
forward has already been used internally, by people who have been using
DB for quite a while (some of us for 20 years). The use of files has
quite a number of advantages, and the project has followed this line of
development, which we question often, but never strong enough as to
abandon current developments and change. The issue of scalability
requires that we look at the DB approach. Having said this, your
forceful approach on DB demands black/white answers (agree/do not agree)
on a subject that for us is much more complex.

If there is change, it will not be tomorow. It requires clear planning
and some security that the new approach is better, which we will only
have through experience. This is why I propose in my prior mail
developing an experimental second DB based back-end (which we are
prepared to fund), to ensure that all data can be easily mapped and that
it works better. If it comes out to be clearly better, we will be the
first ones to go for it.

> I am
>feeling a little frustrated. I sincerely want to help this project as
>much as I can, and I feel that these fundamental issues must be resolved
>before I get deep into the technical details.
>
>
The work that was planned is for your SoC project was very clear, and
does not require any decision on the technology of the back-end. It will
help the implementation of different approaches, but those approaches do
not need to be decided now (even if work on figuring out if they are
better can start immediatly).

You opinion on the back-end is important, as many others, but please
remember that there are other people involved, and that there are
reasons why we do things the way we do them. At some point we might need
to change the way things are made, but we need to be sure that we are
moving to a better approach. Opinions are not enough.

Javier

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST.RemoveThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.RemoveThis@lists.debian.org

Javier SOLA · External Since: Jun 10, 2006 Posts: 5

Gintautas,

You have to talk to Friedle and David, but the idea is to do the
separation on what is there now, using the work and classes that Friedle
and David are doing. They have been putting a lot of work on the base
classes, we should only get into new ones if they are for a different
purpose. As Dwayne mentioned, these clases could easily be adapted later
for other type of storage (the DB), but that is the future, we now
should concentrate on the SoC project.

My impression is that you should now:

1.- Specify what the API should do, what services is should render upwards.
2.- Draw very clearly the line between the present back-end and what is
above it and develop the API from above to the present back-end,
remaking the direct conections through the API.
3.- Document this API, as it should be the same for all possible back-ends.
4.- Write the XML-RPC. This will take some time, as different scenarios
probably need to be considered (if it pushes data or demands data from
other places).

I don't know Python myself, so I cannot help on the details, and maybe
some of the things that I say do not make any sense, I don't know if
what I am saying is in agreement with what you wrote, but I assume that
by "the new back-end" you mean the API, in which case we seem to agree.
If so, I think that we are in the same mind-frame.

But I am just a theory man, at the end the ones that know what is
happening are Dwayne, Friedle and David. I believe that if you work in
close cooperation with them, there will be plenty of time to work on the
separation, API and XML-RPC, and to integrate it in HEAD, so that in
September we are prepared to start thinking about writing and
integrating a database back-end with the same API. I would prefer to
wait until the API is implemented before we start thinking about this,
as specifications may change (be improved) during implementation.

Javier

Gintautas Miliauskas wrote:

>Hello,
>
>
>
>>1.- Gintautas works on the API (which has to include the separation
>>of front-end and back-end), and on the XML-RPC. This will really
>>...
>>
>>
>
>I like your plan very much.
>
>I have one question: how to best approach migration to the new backend?
>Doing it in-place would unnecessarily break a lot of things until the
>entire job is complete. I am thinking of this process:
>
>1) introduce a new Python package, e.g., Pootle.backend or
>Pootle.storage 2) define interfaces (abstract classes)
>3) define implementations, gradually move code into them from the main
>server, update original code to use the new backend
>4) remove all remaining direct access code so that data is accessed
>only through the backend.
>
>Do you think this plan makes sense?
>
>
>
>>Adding a little more meat:... Also, I would see the a separation
>>between the Pootle web-based file server and the Pootle editor, and
>>maybe putting them on top of the API, even if maybe this is not the
>>best place for the editor. Would this break the idea that you
>>expressed with the server layer above?
>>
>><snip>
>>
>>
>
>Yes, this scheme is almost exactly what I was thinking about. Indeed,
>the Pootle file server is different from the Pootle online editor and
>they should be separate.
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Translate-pootle mailing list
>Translate-pootle@lists.sourceforge.net
>https://lists.sourceforge.net/lists/listinfo/translate-pootle
>
>

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Otavio Salvador · External Since: Dec 01, 2004 Posts: 587

Javier SOLA <javier DeleteThis @Khmeros.info> writes:

> Hi Otavio, Guintautas,
>
> I would like to propose a course of action.
>
> 1.- Gintautas works on the API (which has to include the separation of
> front-end and back-end), and on the XML-RPC. This will really clear up
> things for further work, and will facilitate the specification of a
> DB. This work is done in a separate branch, as required by the
> project.

That shpuld be done together with WordForge team since they know the
code better with us and would also help to discover how is the easier
way to split them.

> 2.- The WordForge present staff will integrate these changes in Head
> as soon as they are clear, so that Gintautas' branch and Head stay as
> close as possible, and integration is not a problem later.

Good.

> 3.- Independently of the work that is continuing on WordForge
> (front-end or file-based back-end), WordForge will be happy to fund
> somebody to work on designing and implementing a RDB back-end. This
> can be somebody working on parallel during the coming months, or
> Guintautas after he finishes (if he has time before going back to
> school). Then we just use -or continue working on-the best back-end
> that we have. We would work together with this person to ensure that
> all the data that we need in the project is present in the DB (even if
> most of should be defined already defined by the API).

Yes. I agree that WordForge shouldn't stop its development.

> I somehow have a feeling that the optimal result will be something
> between file-based and DB based, again, I could be completelly
> wrong. We will not really know what it is until we try.

I disagree. I think that the files will be the product of the system.

> Some more comments below.
>
> Otavio Salvador wrote:
>
>>Gintautas Miliauskas <gintas DeleteThis @akl.lt> writes:
>>I fully agree with you that we should avoid to store data directly on
>>files also because if we later discover another format will be very
>>difficult to reuse old data and we'll need to make converters for
>>it. So, let does it from basic design.
>>
>>
> I have to disagree with this. Independently of if the best storage is
> files or DB, the standard formats are precisely done so that they will
> last for a long time. XLIFF is an OASIS standard that will evolve, and
> will not be replaced in the coming decades. Same as Unicode, the
> standards are there to last... and, even if change, happened, you
> would still need to write converters. On the contrary, when these
> standards advance (within backwards compatibility), you will have to
> modify the database to admin new types of information, but you would
> not need to modify a file-based system

Well, it depends. If we use a common database as backend we won't need
to write converters to share translations between all the projects and
formats. We can just reuse a existent string as suggestion in case of
it to be used in more then one project and also, in case of someone
fix the traslation be able to suggest the improvement to other
projects that use the same string.

That wouldn't happen very often but will help or translators a bit.

>>One thing that come to my mind just now is that we might have the
>>following design:
>>
>> |--------------------------------------|
>> | | Pootle itself |
>> | Clients |----------------------------|
>> | | Pootle compatibility layer |
>> |--------------------------------------|
>> | XML-RPC server | Other servers |
>> |--------------------------------------|
>> | API to access the data |
>> |--------------------------------------|
>> | RDMS with data |
>> |--------------------------------------|
>>
>>
>>
> Adding a little more meat:... Also, I would see the a separation
> between the Pootle web-based file server and the Pootle editor, and
> maybe putting them on top of the API, even if maybe this is not the
> best place for the editor. Would this break the idea that you
> expressed with the server layer above?
>
> |-----------------------------------------------------------------------------|
> | Clients | | CVS/SVN |
> | Other Pootle | | FOSS projects |
> | servers | | |
> | Rosetta | Translators Reviewers, etc. | |
> | Off-line Editors| | |
> | Compiler farm | | |
> |-----------------------------------------------------------------------------|
> | XML-RPC server | Pootle file Server| Pootle Editor| Other servers + filters|
> |-----------------------------------------------------------------------------|
> | API to access the data |
> |-----------------------------------------------------------------------------|
> | RDMS with data | File based System | Other back-ends? |
> |-----------------------------------------------------------------------------|

No. It doesn't break anything. When I said Pootle I meant anything
that's involved on it. My suggestion might put Pootle File Server
bellow of Pootle Editor since it might allow us to just put the
compatibility layer at this level.

>>The only drawback on it that I can see is that we won't contribute
>>much code back to Pootle directly. Basically fixes to get it more
>>"backend agnostic" and then plug our backend there. That, IMHO, is a
>>good conseguence since they'll be free to choose and develop other
>>backends without the compromise to us just one.
>>
>>
> Our idea -as I mentioned- is still that we work together in the
> different options. We would not like to see an "our back- end" and a
> "your back-end". The advantage of working together is that there are
> more people thinking, and the good news is that we have sufficient
> funding to look at both options, and then pick up the one that we all
> agree is the best, and continue working in that direction.

My suggestion might, for Gintautas start to work on a prototype of the
RDMS and the API. After we have something useful there, start to write
the Pootle compatibility layer (not the XML-RPC) since it'll be our
first "production system" and also will allow to us to see if it's the
right choice and stop to work on the 'wrong' one ASAP.

Comments?

--
O T A V I O S A L V A D O R
---------------------------------------------
E-mail: otavio DeleteThis @debian.org UIN: 5906116
GNU/Linux User: 239058 GPG ID: 49A5F855
Home Page: http://www.freedom.ind.br/otavio
---------------------------------------------
"Microsoft gives you Windows ... Linux gives
you the whole house."

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Otavio Salvador · External Since: Dec 01, 2004 Posts: 587

Gintautas Miliauskas <gintas.DeleteThis@akl.lt> writes:

> Hello,
>
>> 1.- Gintautas works on the API (which has to include the separation
>> of front-end and back-end), and on the XML-RPC. This will really
>> ...
>
> I like your plan very much.
>
> I have one question: how to best approach migration to the new backend?
> Doing it in-place would unnecessarily break a lot of things until the
> entire job is complete. I am thinking of this process:
>
> 1) introduce a new Python package, e.g., Pootle.backend or
> Pootle.storage 2) define interfaces (abstract classes)
> 3) define implementations, gradually move code into them from the main
> server, update original code to use the new backend
> 4) remove all remaining direct access code so that data is accessed
> only through the backend.
>
> Do you think this plan makes sense?

IMHO, yes.

>> Adding a little more meat:... Also, I would see the a separation
>> between the Pootle web-based file server and the Pootle editor, and
>> maybe putting them on top of the API, even if maybe this is not the
>> best place for the editor. Would this break the idea that you
>> expressed with the server layer above?
>>
>> <snip>
>
> Yes, this scheme is almost exactly what I was thinking about. Indeed,
> the Pootle file server is different from the Pootle online editor and
> they should be separate.

Sure but the file server, IMHO should be bellow so all Pootle tools
can use it and avoid more compatibility layers between all the needed
tools.

--
O T A V I O S A L V A D O R
---------------------------------------------
E-mail: otavio.DeleteThis@debian.org UIN: 5906116
GNU/Linux User: 239058 GPG ID: 49A5F855
Home Page: http://www.freedom.ind.br/otavio
---------------------------------------------
"Microsoft gives you Windows ... Linux gives
you the whole house."

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST.DeleteThis@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.DeleteThis@lists.debian.org

Clytie Siddall · External Since: Apr 29, 2005 Posts: 193

A couple of comments purely from the translator's point of view. Smile

On 10/06/2006, at 8:55 PM, Javier SOLA wrote:
>
> Otavio Salvador wrote:
>
>> Gintautas Miliauskas <gintas.DeleteThis@akl.lt> writes:
>> I fully agree with you that we should avoid to store data directly on
>> files also because if we later discover another format will be very
>> difficult to reuse old data and we'll need to make converters for
>> it. So, let does it from basic design.
>>
> I have to disagree with this. Independently of if the best storage is
> files or DB, the standard formats are precisely done so that they will
> last for a long time. XLIFF is an OASIS standard that will evolve, and
> will not be replaced in the coming decades. Same as Unicode, the
> standards are there to last...

and in fact will _save_ us further conversion and fiddling around, as
they are used more. Among other things, use of the standards will
facilitate co-operation between the professional and free-software
translation arenas. They are a basis for effective growth and
communication in the likely future.

> and, even if change, happened, you would
> still need to write converters. On the contrary, when these standards
> advance (within backwards compatibility), you will have to modify the
> database to admin new types of information, but you would not need to
> modify a file-based system

The file-based system using XLIFF can also allow for deeper and more
direct interaction between the infrastructure and the translator/
language team.
>
>> One thing that come to my mind just now is that we might have the
>> following design:
>>
>> |--------------------------------------|
>> | | Pootle itself |
>> | Clients |----------------------------|
>> | | Pootle compatibility layer |
>> |--------------------------------------|
>> | XML-RPC server | Other servers |
>> |--------------------------------------|
>> | API to access the data |
>> |--------------------------------------|
>> | RDMS with data |
>> |--------------------------------------|
>>
>>
> Adding a little more meat:... Also, I would see the a separation
> between
> the Pootle web-based file server and the Pootle editor,

This separation can also allow for different editors to interact with
the file server. I know LocFactoryEditor's developer is interested in
interacting with the Pootle server, and I think the Kbabel developers
would also be interested. This type of pluggability will make it much
easier for translators to switch between online and offline editing,
depending on the situation.

[File menu]
New
Open...
Open from server...
....
Save
Save to server...

just as a good text editor interacts with FTP servers and source
control.

> and maybe
> putting them on top of the API, even if maybe this is not the best
> place
> for the editor. Would this break the idea that you expressed with the
> server layer above?

and I see "Offline Editors" in the diagram below. This means the
interaction between the main server layer and the one above has to be
flexible enough to allow for different workflows. (I know this kind
of detail isn't that relevant yet, but it's worth keeping in mind
while working on the main server layer at least.)
>
>
> |---------------------------------------------------------------------
> --------|
> | Clients | | CVS/
> SVN |
> | Other Pootle | | FOSS
> projects |
> | servers |
> | |
> | Rosetta | Translators Reviewers, etc.
> | |
> | Off-line Editors|
> | |
> | Compiler farm |
> | |
>
> |---------------------------------------------------------------------
> --------|
> | XML-RPC server | Pootle file Server| Pootle Editor| Other
> servers + filters|
>
> |---------------------------------------------------------------------
> --------|
> | API to access the
> data |
>
> |---------------------------------------------------------------------
> --------|
> | RDMS with data | File based System | Other back-
> ends? |
>
> |---------------------------------------------------------------------
> --------|
>
>
>> The only drawback on it that I can see is that we won't contribute
>> much code back to Pootle directly. Basically fixes to get it more
>> "backend agnostic" and then plug our backend there. That, IMHO, is a
>> good conseguence since they'll be free to choose and develop other
>> backends without the compromise to us just one.
>>
>>
> Our idea -as I mentioned- is still that we work together in the
> different options. We would not like to see an "our back- end" and a
> "your back-end". The advantage of working together is that there are
> more people thinking, and the good news is that we have sufficient
> funding to look at both options, and then pick up the one that we all
> agree is the best, and continue working in that direction.
>
If we build into the system a capacity to interact with different
backends, nobody will be locked into any single choice in the future.
At different times (and for different projects), different choices
will be appropriate. Keeping our options open at both backend and
frontend interaction levels reduces the barriers not only for
translator and language-team participation, but also for ongoing
technical contribution from the free-software community. I don't mean
that people will be running up new backends over the weekend, but
that whatever situation comes up, people will have the option to
adapt current backends, or to take advantage of new opportunities.

Not so much "agnostic" as "aware".

from Clytie (vi-VN, Vietnamese free-software translation team / nhóm
Việt hóa phần mềm tự do)
http://groups-beta.google.com/group/vi-VN

Javier SOLA · External Since: Jun 10, 2006 Posts: 5

Otavio, Gintautas,

I think that we are rolling now, except that, if we want to start with
the DB as soon as possible, somebody else -other than Gintautas, who
will be busy with the API- should start looking into the design of the
DB, and then adapt to the API that Gintautas will develop. It would be
great if we can find somebody who has the right expertise and is
already involved in Debian. As I said, WorForge could put some money
into it. We would work with this developer providing information and
guidance (types of information and relationships) and Dwayne also has
quite a bit of experience on RDB design. The developer would have to
work closely with Gintautas, as while doing the DB design, a lot of new
necessary pieces of data will come out (metadata that is not in the
XLIFF files about workflow, statistics, etc...) and this data will also
have to be managed by the API.

The API will have to include things that are not implemented now,
building on the work that Friedle is doing on statistics, as we will
have to keep data on goals, as well as objects that we have not yet
implemtented (data on projects, packages..). I would be really happy if
we have this at the end of the summer. It will take work from many of
us, but will clearly finish the design of the data that will be used by
Pootle at the end

I agree that this work should take priority over the XML-RPC.

If you agree with this. How do we go about finding somebody who has a
couple of months to work on the DB?

Some more comments below.

>If we use a common database as backend we won't need
>to write converters to share translations between all the projects and
>formats. We can just reuse a existent string as suggestion in case of
>it to be used in more then one project and also, in case of someone
>fix the traslation be able to suggest the improvement to other
>projects that use the same string.
>
>That wouldn't happen very often but will help or translators a bit.
>
>
Translation memory would always be in a database (capable of importing
and exporting TMX format), which would be common to all projects in the
same language, even if each translation would be marked with its project
and string of origin, so that matching engines can give priority to
strings that come from the same project, but can also find others. The
translation memory database would be (logically) separate from the DB
that includes current projects, as it might come from the outside (from
projects external to Pootle) or match translation that are no longer in
this Pootle server.

Nevertheless, if all was in databases, the process of matching
translation memory would have as result that information is copied from
the TM DB into the "current projects" database, as matches that are
related to a particular untranslated string.

>>Also, I would see the a separation
>>between the Pootle web-based file server and the Pootle editor, and
>>maybe putting them on top of the API, even if maybe this is not the
>>best place for the editor. Would this break the idea that you
>>expressed with the server layer above?
>>
>>|-----------------------------------------------------------------------------|
>>| Clients | | CVS/SVN |
>>| Other Pootle | | FOSS projects |
>>| servers | | |
>>| Rosetta | Translators Reviewers, etc. | |
>>| Off-line Editors| | |
>>| Compiler farm | | |
>>|-----------------------------------------------------------------------------|
>>| XML-RPC server | Pootle file Server| Pootle Editor| Other servers + filters|
>>|-----------------------------------------------------------------------------|
>>| API to access the data |
>>|-----------------------------------------------------------------------------|
>>| RDMS with data | File based System | Other back-ends? |
>>|-----------------------------------------------------------------------------|
>>
>>
>
>No. It doesn't break anything. When I said Pootle I meant anything
>that's involved on it. My suggestion might put Pootle File Server
>bellow of Pootle Editor since it might allow us to just put the
>compatibility layer at this level.
>
>
This is probably correct. At the end the editor will have to work with
files. In the Pootle server interface, the translator will first
identify the files that (s)he wants to work in, and then will decide if
(s)he want to translate them online or download them. The editor is
called from the server. I would like to have Friedle's or David's
opinion on this.

I will copy this design to the Wiki, putting the editor on top of Pootle.

Javier

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Javier SOLA · External Since: Jun 10, 2006 Posts: 5

Gintautas Miliauskas wrote:

>Hello,
>
>I've started sketching the API, but I would like to have a chat on IRC
>with the developers who are working on the new base classes. Should I
>use your branch or start a new one? When can I find you online, guys?
>Perhaps I can find you on Jabber? Ottavio? David? Friedle?
>
>
>
>>I think that we are rolling now, except that, if we want to start
>>with the DB as soon as possible, somebody else -other than Gintautas,
>>who will be busy with the API- should start looking into the design
>>of the DB, and then adapt to the API that Gintautas will develop. It
>>would be great if we can find somebody who has the right expertise
>>and is already involved in Debian.
>>
>>
>I would suggest to wait until the API works with the file-based backend
>before starting work on an RDB binding.
>
>Designing the DB schema should not be difficult here, because we will
>have the object model in the API and we will only need to do a direct
>translation "OO model" -> "RDB schema", which is a mechanical task.
>What is needed here is experience of engineering components that use an
>RDB, and admittedly I don't have much of that.
>
>
Otavio, what is your opinion on this?

>>This is probably correct. At the end the editor will have to work
>>with files. In the Pootle server interface, the translator will first
>>identify the files that (s)he wants to work in, and then will decide
>>if (s)he want to translate them online or download them. The editor
>>is called from the server. I would like to have Friedle's or David's
>>opinion on this.
>>
>>
>I'm not Friedle or David Wink

but I think that this is not conceptually
>sound. In the online editor the user is not editing a "file", s/he is
>editing translations. That's the whole point: if they want a file,
>they use the "file server" frontend, and if they want to change a single
>string through-the-web, they use a web-based frontend built directly on
>our backend which will have a specific operation for changing a single
>string. Would you agree?
>
>
We have to assume that contextual information in necessary. If you are
translating an OpenOffice help file, you really want to know what are
the prior and posterior strings are. You can either get them from the
DB, or just use the API access of the server, which is already there...
and use complete files. The present system uses complete files, and it
would probably be a lot of change to have to move this to direct API
access. I think that Otavio is right on this one... but let's ask Friedle.

Javier

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST.TakeThisOut@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster.TakeThisOut@lists.debian.org

David Fraser · External Since: May 25, 2006 Posts: 3

Gintautas Miliauskas wrote:
> Hello,
>
> I've started sketching the API, but I would like to have a chat on IRC
> with the developers who are working on the new base classes. Should I
> use your branch or start a new one? When can I find you online, guys?
> Perhaps I can find you on Jabber? Ottavio? David? Friedle?
>
Saw you on IRC over the weekend but unfortunately couldn't chat. #pootle
on irc.freenode.net is where we usually hang out
My jabber address is david at jabber.obsidian.co.za
I think the first thing is for you to look at the current base classes
and API, and then we can discuss how it works (I think you read some of
the stuff on the wiki... toolkit/base classes)
translate toolkit HEAD branch is the place to look, the underlying
formats are in /translate/storage
there is a separate Pootle branch that I've been using to migrate
towards using the base API (rather than the old PO class API) and
locking, called Pootle-locking-branch
There is a page on that in the wiki too (pootle/locking)
>> I think that we are rolling now, except that, if we want to start
>> with the DB as soon as possible, somebody else -other than Gintautas,
>> who will be busy with the API- should start looking into the design
>> of the DB, and then adapt to the API that Gintautas will develop. It
>> would be great if we can find somebody who has the right expertise
>> and is already involved in Debian.
>>
> I would suggest to wait until the API works with the file-based backend
> before starting work on an RDB binding.
>
> Designing the DB schema should not be difficult here, because we will
> have the object model in the API and we will only need to do a direct
> translation "OO model" -> "RDB schema", which is a mechanical task.
> What is needed here is experience of engineering components that use an
> RDB, and admittedly I don't have much of that.
>
Lets discuss this on IRC as well.
>> information and guidance (types of information and relationships) and
>> Dwayne also has quite a bit of experience on RDB design. The
>> developer would have to work closely with Gintautas, as while doing
>> the DB design, a lot of new necessary pieces of data will come out
>> (metadata that is not in the XLIFF files about workflow, statistics,
>> etc...) and this data will also have to be managed by the API.
>>
> Everything that is in XLIFF will have to be in the API too, because it
> is an abstraction layer. There can't possibly be any piece of
> information that is not abstracted, so there should not be anything new
> directly related to an RDB.
>
One of the issues is that XLIFF being based on XML is hierarchical, and
how best to represent that in a relational database. There's plenty to
talk about here and its not neccessarily straightforward. I agree with
the approach suggested, to separate out a DB backend construction, asd
it enables us to keep moving on the current tasks.
One possibility is to implement a simple DB backend that doesn't have
the complexity of XLIFF... but again, lets discuss this.
>> The API will have to include things that are not implemented now,
>> building on the work that Friedle is doing on statistics, as we will
>> have to keep data on goals, as well as objects that we have not yet
>> implemtented (data on projects, packages..). I would be really happy
>> if we have this at the end of the summer. It will take work from many
>> of us, but will clearly finish the design of the data that will be
>> used by Pootle at the end
>>
> Sure, statistics will be made abstract operations too.
>
>> I agree that this work should take priority over the XML-RPC.
>>
> XML-RPC would be a layer above the new backend, so it makes no sense to
> start doing it now.
>
Agreed. In fact once Pootle interacts with the storage using the Python
API, XML-RPC services could be offered alongside that but Pootle doesn't
*have* to use them (it may just slow things down talking XML-RPC to the
same machine) - although it would be very useful for scalability and
distributing options, so should be supported
>> but can also find others. The translation memory database would be
>> (logically) separate from the DB that includes current projects, as
>> it might come from the outside (from projects external to Pootle) or
>> match translation that are no longer in this Pootle server.
>>
> IOW, the translation memory should be a separate component. Agreed.
>
With translation memory again text indexing is what's really useful...
>> This is probably correct. At the end the editor will have to work
>> with files. In the Pootle server interface, the translator will first
>> identify the files that (s)he wants to work in, and then will decide
>> if (s)he want to translate them online or download them. The editor
>> is called from the server. I would like to have Friedle's or David's
>> opinion on this.
>>
> I'm not Friedle or David Wink

but I think that this is not conceptually
> sound. In the online editor the user is not editing a "file", s/he is
> editing translations. That's the whole point: if they want a file,
> they use the "file server" frontend, and if they want to change a single
> string through-the-web, they use a web-based frontend built directly on
> our backend which will have a specific operation for changing a single
> string. Would you agree?
>
The way Pootle currently works you can have it either way - you can be
working project wide or in a specific directory or file, from there you
can do searches or translate untranslated strings etc etc
It is important though that if you are doing string by string operation
the order in the files is preserved as there is a logical continuity often

Sorry I have been dipping in and out of this conversation and have much
more I could say but others are saying plenty Smile

Just for reference must of my work is with relational databases so I do
understand their advantages (hence the pootle/metadata page on the
wiki). Keeping going with working with files directly up till now has
enabled us to handle some of the complexity, which we will need to
continue doing as we move ahead...

Cheers
David

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST DeleteThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster DeleteThis @lists.debian.org

Otavio Salvador · External Since: Dec 01, 2004 Posts: 587

Gintautas Miliauskas <gintas RemoveThis @akl.lt> writes:

> Hello,
>
> I've started sketching the API, but I would like to have a chat on IRC
> with the developers who are working on the new base classes. Should I
> use your branch or start a new one? When can I find you online, guys?
> Perhaps I can find you on Jabber? Ottavio? David? Friedle?

otavio at jabber.org

>> I think that we are rolling now, except that, if we want to start
>> with the DB as soon as possible, somebody else -other than Gintautas,
>> who will be busy with the API- should start looking into the design
>> of the DB, and then adapt to the API that Gintautas will develop. It
>> would be great if we can find somebody who has the right expertise
>> and is already involved in Debian.
>
> I would suggest to wait until the API works with the file-based backend
> before starting work on an RDB binding.

I agree. Doing it in a step by step is easier also because we can
then merge it directly on Pootle and free them to continue to improve it.

> Designing the DB schema should not be difficult here, because we will
> have the object model in the API and we will only need to do a direct
> translation "OO model" -> "RDB schema", which is a mechanical task.
> What is needed here is experience of engineering components that use an
> RDB, and admittedly I don't have much of that.

What's the difference of to buld a component that will use a file or a RDB?

> I'm not Friedle or David Wink

but I think that this is not conceptually
> sound. In the online editor the user is not editing a "file", s/he is
> editing translations. That's the whole point: if they want a file,
> they use the "file server" frontend, and if they want to change a single
> string through-the-web, they use a web-based frontend built directly on
> our backend which will have a specific operation for changing a single
> string. Would you agree?

In general case, yes. Currently the Pootle will always need to use the
File server or will be A LOT of work to change it to use the right
layers directly. That was my idea to be easier to allow Pootle to use
the new backend without too many changes.

--
O T A V I O S A L V A D O R
---------------------------------------------
E-mail: otavio RemoveThis @debian.org UIN: 5906116
GNU/Linux User: 239058 GPG ID: 49A5F855
Home Page: http://www.freedom.ind.br/otavio
---------------------------------------------
"Microsoft gives you Windows ... Linux gives
you the whole house."

--
To UNSUBSCRIBE, email to debian-i18n-REQUEST RemoveThis @lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmaster RemoveThis @lists.debian.org

Clytie Siddall · External Since: Apr 29, 2005 Posts: 193

On 11/06/2006, at 8:33 PM, Javier SOLA wrote:

> We have to assume that contextual information in necessary. If you
> are translating an OpenOffice help file, you really want to know
> what are the prior and posterior strings are. You can either get
> them from the DB, or just use the API access of the server, which
> is already there... and use complete files. The present system uses
> complete files, and it would probably be a lot of change to have to
> move this to direct API access. I think that Otavio is right on
> this one... but let's ask Friedle.

Please assume contextual information is necessary and often essential.

I have noticed that the quality of translations is so often
proportional to the amount of available context. The optimum
situation is where the translator is an experienced user of that
software, but you can't be an experienced user of everything you
translate. You rely on the available context. With the new gettext
version: (from the NEWS file):

> * GUI program support:
> - PO files can now contain messages constrained to a certain
> context.
> Most often such a context is a menu, dialog or panel
> identification.
> The syntax in the PO file is
> msgctxt "context"
> msgid "original"
> msgstr "translation"
> - The xgettext program can be told through the --keyword flag which
> function/macro argument has the role of a context.
> - The (non-public) include file gettext.h defines macros
> pgettext, dpgettext
> etc. that take a context argument.
> For more information, see the node "Contexts" in the manual.

we have the opportunity to manipulate context more successfully.

I think lack of context has been the single most frustrating
situation I have encountered in software translation. Most files have
no contextual information at all, bar comparison with other strings.
In the huge GIMP main translation file, I was presented with the string:

msgid "S"
msgstr ""

There was no context. None of the strings in that section of the file
turned out to be related to it, or to provide any clue to its
meaning. When I queried the developer, he told me, "You should know
what it means."

Being presented with one string alone, even if it has more than one
character, is often of very little use. The Google translation
interface is an effective example of this problem. You are presented
with one single string, and the "Help" (context) field is usually empty.

By contrast, the Debian-Installer Level 1 file has the best
contextual support I have seen across the free software projects. I
would suggest it as a model when determining how to process context.

from Clytie (vi-VN, Vietnamese free-software translation team / nhóm
Việt hóa phần mềm tự do)
http://groups-beta.google.com/group/vi-VN