[Yt-dev] Parameter file store

[Matthew Turk]

Hi Dave,

Here's a patch:

http://paste.enzotools.org/show/68/

You can get this with yt_lodgeit.py --download=68 > patch and apply it
with patch -p1 < patch .  Conversely, you can download it from the
bitbucket repo or the mirror on hg.enzotools.org.  It defaults to
keeping the last 300 touched files around.

If this works I'll merge back to trunk.  You can test it just by
instantiated a crapton of them in a row.

-Matt

On Tue, Mar 3, 2009 at 2:25 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> That's right, it's completely re-creatable.
>
> On Tue, Mar 3, 2009 at 2:19 PM, david collins <antpuncher at gmail.com> wrote:
>> Sure.  So one would need to re-instantiate if something gets bumped off?
>>
>> d.
>>
>> On Tue, Mar 3, 2009 at 2:15 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>> Okay, how about this.
>>>
>>> We make a 'freshness' column in the CSV.  This is a timestamp of the
>>> last time the parameter file was used.  This can be the sort field,
>>> too.  We'll set a variable -- default it to, say, 300 -- that includes
>>> the freshness cutoff in number of parameter files.  Thus, sorted by
>>> freshness, we take the last N, where N is set in the parameter file.
>>>
>>> Sound good?
>>>
>>> On Tue, Mar 3, 2009 at 1:47 PM, david collins <antpuncher at gmail.com> wrote:
>>>>> Wow, you've been analyzing 10,000 parameter files?  Okay.
>>>>
>>>> Ci.  Making time sequences.  Hundreds/thousands of snapshots from a
>>>> dozen sims, and comparing.   (This is why uber)
>>>>
>>>> Deleting parameter_files.csv is a good workaround for me now, so don't
>>>> spend more than 45 seconds on it.  I did this recently, so it's not
>>>> huge, only 68 Mb.
>>>>
>>>> kraken /nics/b/home/collins/.yt/parameter_files.csv
>>>>
>>>> d.
>>>>
>>>>>
>>>>> One thing I can do is implement a maximum number in the store.  Then
>>>>> we can only guarantee objects will be automatically retrieved (without
>>>>> having *recently* instantiated the PF) if they were in the last N,
>>>>> where N is the number of files we keep in the store.
>>>>>
>>>>> Can you send me one of your parameter file stores?  Or put it on kraken?
>>>>>
>>>>> -Matt
>>>>> - Show quoted text -
>>>>> On Tue, Mar 3, 2009 at 1:04 PM, david collins <antpuncher at gmail.com> wrote:
>>>>>> Hey--
>>>>>>
>>>>>> Not sure if this belongs on the dev list--
>>>>>>
>>>>>> I've been reading and analyzing large-ish ( O(10,000) ) parameter
>>>>>> files, and the parameter_file.csv gets large (300 Mb), which isn't a
>>>>>> problem, but when that happens, doing anything (even importing YT)
>>>>>> takes forever (> 1 minute to import yt.lagos).
>>>>>>
>>>>>> I did install the "unboudned growth" fix you sent, still happens.
>>>>>>
>>>>>> I just noticed that it  happens after your fix yesterday, or I'da said
>>>>>> something earlier.
>>>>>>
>>>>>> d.
>>>>>>
>>>>> _______________________________________________
>>>>> Yt-dev mailing list
>>>>> Yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>
>>>> _______________________________________________
>>>> Yt-dev mailing list
>>>> Yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>> _______________________________________________
>>> Yt-dev mailing list
>>> Yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>> _______________________________________________
>> Yt-dev mailing list
>> Yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
>