[Yt-dev] Parameter file store

david collins antpuncher at gmail.com
Tue Mar 3 14:19:50 PST 2009


Sure.  So one would need to re-instantiate if something gets bumped off?

d.

On Tue, Mar 3, 2009 at 2:15 PM, Matthew Turk <matthewturk at gmail.com> wrote:
> Okay, how about this.
>
> We make a 'freshness' column in the CSV.  This is a timestamp of the
> last time the parameter file was used.  This can be the sort field,
> too.  We'll set a variable -- default it to, say, 300 -- that includes
> the freshness cutoff in number of parameter files.  Thus, sorted by
> freshness, we take the last N, where N is set in the parameter file.
>
> Sound good?
>
> On Tue, Mar 3, 2009 at 1:47 PM, david collins <antpuncher at gmail.com> wrote:
>>> Wow, you've been analyzing 10,000 parameter files?  Okay.
>>
>> Ci.  Making time sequences.  Hundreds/thousands of snapshots from a
>> dozen sims, and comparing.   (This is why uber)
>>
>> Deleting parameter_files.csv is a good workaround for me now, so don't
>> spend more than 45 seconds on it.  I did this recently, so it's not
>> huge, only 68 Mb.
>>
>> kraken /nics/b/home/collins/.yt/parameter_files.csv
>>
>> d.
>>
>>>
>>> One thing I can do is implement a maximum number in the store.  Then
>>> we can only guarantee objects will be automatically retrieved (without
>>> having *recently* instantiated the PF) if they were in the last N,
>>> where N is the number of files we keep in the store.
>>>
>>> Can you send me one of your parameter file stores?  Or put it on kraken?
>>>
>>> -Matt
>>> - Show quoted text -
>>> On Tue, Mar 3, 2009 at 1:04 PM, david collins <antpuncher at gmail.com> wrote:
>>>> Hey--
>>>>
>>>> Not sure if this belongs on the dev list--
>>>>
>>>> I've been reading and analyzing large-ish ( O(10,000) ) parameter
>>>> files, and the parameter_file.csv gets large (300 Mb), which isn't a
>>>> problem, but when that happens, doing anything (even importing YT)
>>>> takes forever (> 1 minute to import yt.lagos).
>>>>
>>>> I did install the "unboudned growth" fix you sent, still happens.
>>>>
>>>> I just noticed that it  happens after your fix yesterday, or I'da said
>>>> something earlier.
>>>>
>>>> d.
>>>>
>>> _______________________________________________
>>> Yt-dev mailing list
>>> Yt-dev at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>>
>> _______________________________________________
>> Yt-dev mailing list
>> Yt-dev at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>>
> _______________________________________________
> Yt-dev mailing list
> Yt-dev at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org
>



More information about the yt-dev mailing list