[yt-users] .yt file

[Matthew Turk]

Hi Mike,

On Thu, Dec 9, 2010 at 9:03 PM, Michael Kuhlen <mqk at astro.berkeley.edu> wrote:
>> I see that the file is in HDF5 format. Would it be safe to just delete
>> the one offending dataset from it?
>
> It seems the answer to this is "yes". At least it appears to be working
> for me.
>
> It just occurred to me that this issue will arise more generically (not
> just in the case of a bug), when the user-defined derived field depends
> on some externally supplied parameters.

Yes, this is correct.  At the risk of writing what has been charmingly
referred to as a Mattifesto, I'll give a bit of background on the
topic.

The idea of serializing projections, in between runs, was to avoid
having to perform a potentially expensive operation over and over --
in the time since, I've come to reconsider the wisdom of implicitly
serializing projections.  Originally, fields were pretty simple, and
you would just project, say, Density.  (This was even before weighting
was available, so it was only line integrals.)

In the intervening time, it has become more easy and possible to
create relatively advanced fields.  The logic for deciding on when to
serialize has also become much more complex, and a non-negligible
portion of overhead to modifying the AMRProjBase code is ensuring that
previous serialization behavior is not broken.  One of the biggest
issues is determining what fully determines a projected field, and how
we can check that it's constant between runs: it's a combination of
the simulation output (checked), the axis (checked), the weighting
(checked) the data source, if any cuts are applied (checked), and the
input to the field generation (not checked).  Unfortunately, you are
correct: if you have, for instance, a field that accepts a bulk
velocity, or an angular momentum vector (as I have recently done) you
will run into issues, because modifying that parameter does not
currently signal to the serialization mechanism that the field needs
to be regenerated.

This is, as you note, really unfortunate.  There are two ways I see of
preventing this: we can either reject the serialization of any field
that requires a field parameter (I don't know if this option is really
feasible) or we can turn off global, automatic serialization of
projections.  The latter option is one that has crossed my mind a lot
recently, but I suspect that because I deal with smaller simulation
outputs compared to some of the bigger cosmology folks in the crowd
here today, my viewpoint is not the one that we should look for that.

But, there are other options available, which require manual
intervention, that will disable serialization.  (The deletion of
datasets, as I'm sure you noticed, won't actually reduce the .yt file
in size unless you repack it.)  You can specify serialize=False to the
construction of an AMRProj, but this won't work through the
PlotCollection interface.  Alternately, you can, at the top of your
script, import ytcfg from yt.config and set
ytcfg["yt","serialize"]="False" or
ytcfg["yt","onlydeserialize"]="True".

What are your thoughts on all this?  Do you think that this is a
problem we should aggressively address?  It's not clear to me that
very many field parameter-dependent projections are being made at this
time, but it is worrisome that it has caused you annoyance.  Does
anyone else have any ideas/comments?

Best,

Matt

>
> Mike
>
> On 12/09/2010 08:36 PM, Michael Kuhlen wrote:
>> Hi
>>
>> I'm experimenting with a new user defined derived quantity, and
>> occasionally I've used a buggy version of this to create a projection
>> (say), which then got saved to the .yt file. After fixing the bug in the
>> definition of my derived quantity, I'd like to try out the new version,
>> but of course yt will read the old buggy version from the .yt file. I
>> could just delete the .yt file, but it's storing a lot of other correct
>> projections and I'd like to avoid re-running these.
>>
>> Is there some keyword to force yt to skip reading from the .yt file?
>> And/or one to force it not to write to the .yt in the first place?
>>
>> I see that the file is in HDF5 format. Would it be safe to just delete
>> the one offending dataset from it?
>>
>> Mike
>>
>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
> --
> *********************************************************************
> *                                                                   *
> *  Dr. Michael Kuhlen              Theoretical Astrophysics Center  *
> *  email: mqk at astro.berkeley.edu   UC Berkeley                      *
> *  cell phone: (831) 588-1468      601 Campbell Hall                *
> *  skype username: mikekuhlen      Berkeley, CA 94720               *
> *                                                                   *
> *********************************************************************
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>