[Yt-dev] Parameter file store

[david collins]

> So no problems?  I should feel free to apply the patch to trunk?

Seems good so far.  I re-read and re-plotted some clumps I made a
while ago, and it matches a plot I made about a week ago.  Patch away.

d.


>
> On Fri, Mar 6, 2009 at 9:32 AM, david collins <antpuncher at gmail.com> wrote:
>> Dude.
>>
>> Looking at time per parameter file read, reading the first 600 steps
>> from one sim.
>>
>> Before the fix, it monotonically increased from 0.01 second/read to 3
>> seconds/read.
>>
>> After the fix, it's roughly constant at 0.02.  Huge improvement in
>> performance.  That's really going to speed things up when I make my
>> core tracking analysis.  Thanks!
>>
>> d.
>>
>> On Thu, Mar 5, 2009 at 10:49 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>> Any luck?
>>>
>>> On Tue, Mar 3, 2009 at 3:19 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>> Hi Dave,
>>>>
>>>> Here's a patch:
>>>>
>>>> http://paste.enzotools.org/show/68/
>>>>
>>>> You can get this with yt_lodgeit.py --download=68 > patch and apply it
>>>> with patch -p1 < patch .  Conversely, you can download it from the
>>>> bitbucket repo or the mirror on hg.enzotools.org.  It defaults to
>>>> keeping the last 300 touched files around.
>>>>
>>>> If this works I'll merge back to trunk.  You can test it just by
>>>> instantiated a crapton of them in a row.
>>>>
>>>> -Matt
>>>>
>>>> On Tue, Mar 3, 2009 at 2:25 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>> That's right, it's completely re-creatable.
>>>>>
>>>>> On Tue, Mar 3, 2009 at 2:19 PM, david collins <antpuncher at gmail.com> wrote:
>>>>>> Sure.  So one would need to re-instantiate if something gets bumped off?
>>>>>>
>>>>>> d.
>>>>>>
>>>>>> On Tue, Mar 3, 2009 at 2:15 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>>>>>> Okay, how about this.
>>>>>>>
>>>>>>> We make a 'freshness' column in the CSV.  This is a timestamp of the
>>>>>>> last time the parameter file was used.  This can be the sort field,
>>>>>>> too.  We'll set a variable -- default it to, say, 300 -- that includes
>>>>>>> the freshness cutoff in number of parameter files.  Thus, sorted by
>>>>>>> freshness, we take the last N, where N is set in the parameter file.
>>>>>>>
>>>>>>> Sound good?
>>>>>>>
>>>>>>> On Tue, Mar 3, 2009 at 1:47 PM, david collins <antpuncher at gmail.com> wrote:
>>>>>>>>> Wow, you've been analyzing 10,000 parameter files?  Okay.
>>>>>>>>
>>>>>>>> Ci.  Making time sequences.  Hundreds/thousands of snapshots from a
>>>>>>>> dozen sims, and comparing.   (This is why uber)
>>>>>>>>
>>>>>>>> Deleting parameter_files.csv is a good workaround for me now, so don't
>>>>>>>> spend more than 45 seconds on it.  I did this recently, so it's not
>>>>>>>> huge, only 68 Mb.
>>>>>>>>
>>>>>>>> kraken /nics/b/home/collins/.yt/parameter_files.csv
>>>>>>>>
>>>>>>>> d.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> One thing I can do is implement a maximum number in the store.  Then
>>>>>>>>> we can only guarantee objects will be automatically retrieved (without
>>>>>>>>> having *recently* instantiated the PF) if they were in the last N,
>>>>>>>>> where N is the number of files we keep in the store.
>>>>>>>>>
>>>>>>>>> Can you send me one of your parameter file stores?  Or put it on kraken?
>>>>>>>>>
>>>>>>>>> -Matt
>>>>>>>>> - Show quoted text -
>>>>>>>>> On Tue, Mar 3, 2009 at 1:04 PM, david collins <antpuncher at gmail.com> wrote:
>>>>>>>>>> Hey--
>>>>>>>>>>
>>>>>>>>>> Not sure if this belongs on the dev list--
>>>>>>>>>>
>>>>>>>>>> I've been reading and analyzing large-ish ( O(10,000) ) parameter
>>>>>>>>>> files, and the parameter_file.csv gets large (300 Mb), which isn't a
>>>>>>>>>> problem, but when that happens, doing anything (even importing YT)
>>>>>>>>>> takes forever (> 1 minute to import yt.lagos).
>>>>>>>>>>
>>>>>>>>>> I did install the "unboudned growth" fix you sent, still happens.
>>>>>>>>>>
>>>>>>>>>> I just noticed that it  happens after your fix yesterday, or I'da said
>>>>>>>>>> something earlier.
>>>>>>>>>>
>>>>>>>>>> d.
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Yt-dev mailing list
>>>>>>>>> Yt-dev at lists.spacepope.org
>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Yt-dev mailing list
>>>>>>>> Yt-dev at lists.spacepope.org
>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Yt-dev mailing list
>>>>>>> Yt-dev at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Yt-dev mailing list
>>>>>> Yt-dev at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> Yt-dev mailing list
>>> Yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>> _______________________________________________
>> Yt-dev mailing list
>> Yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>