[Yt-dev] Parameter file store

[Matthew Turk]

Any luck?

On Tue, Mar 3, 2009 at 3:19 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> Hi Dave,
>
> Here's a patch:
>
> http://paste.enzotools.org/show/68/
>
> You can get this with yt_lodgeit.py --download=68 > patch and apply it
> with patch -p1 < patch .  Conversely, you can download it from the
> bitbucket repo or the mirror on hg.enzotools.org.  It defaults to
> keeping the last 300 touched files around.
>
> If this works I'll merge back to trunk.  You can test it just by
> instantiated a crapton of them in a row.
>
> -Matt
>
> On Tue, Mar 3, 2009 at 2:25 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>> That's right, it's completely re-creatable.
>>
>> On Tue, Mar 3, 2009 at 2:19 PM, david collins <antpuncher at gmail.com> wrote:
>>> Sure.  So one would need to re-instantiate if something gets bumped off?
>>>
>>> d.
>>>
>>> On Tue, Mar 3, 2009 at 2:15 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>> Okay, how about this.
>>>>
>>>> We make a 'freshness' column in the CSV.  This is a timestamp of the
>>>> last time the parameter file was used.  This can be the sort field,
>>>> too.  We'll set a variable -- default it to, say, 300 -- that includes
>>>> the freshness cutoff in number of parameter files.  Thus, sorted by
>>>> freshness, we take the last N, where N is set in the parameter file.
>>>>
>>>> Sound good?
>>>>
>>>> On Tue, Mar 3, 2009 at 1:47 PM, david collins <antpuncher at gmail.com> wrote:
>>>>>> Wow, you've been analyzing 10,000 parameter files?  Okay.
>>>>>
>>>>> Ci.  Making time sequences.  Hundreds/thousands of snapshots from a
>>>>> dozen sims, and comparing.   (This is why uber)
>>>>>
>>>>> Deleting parameter_files.csv is a good workaround for me now, so don't
>>>>> spend more than 45 seconds on it.  I did this recently, so it's not
>>>>> huge, only 68 Mb.
>>>>>
>>>>> kraken /nics/b/home/collins/.yt/parameter_files.csv
>>>>>
>>>>> d.
>>>>>
>>>>>>
>>>>>> One thing I can do is implement a maximum number in the store.  Then
>>>>>> we can only guarantee objects will be automatically retrieved (without
>>>>>> having *recently* instantiated the PF) if they were in the last N,
>>>>>> where N is the number of files we keep in the store.
>>>>>>
>>>>>> Can you send me one of your parameter file stores?  Or put it on kraken?
>>>>>>
>>>>>> -Matt
>>>>>> - Show quoted text -
>>>>>> On Tue, Mar 3, 2009 at 1:04 PM, david collins <antpuncher at gmail.com> wrote:
>>>>>>> Hey--
>>>>>>>
>>>>>>> Not sure if this belongs on the dev list--
>>>>>>>
>>>>>>> I've been reading and analyzing large-ish ( O(10,000) ) parameter
>>>>>>> files, and the parameter_file.csv gets large (300 Mb), which isn't a
>>>>>>> problem, but when that happens, doing anything (even importing YT)
>>>>>>> takes forever (> 1 minute to import yt.lagos).
>>>>>>>
>>>>>>> I did install the "unboudned growth" fix you sent, still happens.
>>>>>>>
>>>>>>> I just noticed that it  happens after your fix yesterday, or I'da said
>>>>>>> something earlier.
>>>>>>>
>>>>>>> d.
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Yt-dev mailing list
>>>>>> Yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>
>>>>> _______________________________________________
>>>>> Yt-dev mailing list
>>>>> Yt-dev at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>
>>>> _______________________________________________
>>>> Yt-dev mailing list
>>>> Yt-dev at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>
>>> _______________________________________________
>>> Yt-dev mailing list
>>> Yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>>
>