[yt-users] How to fully close a database

[Nathan Goldbaum]

On Tue, Feb 7, 2017 at 1:42 PM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:

>
>
> On Tue, Feb 7, 2017 at 1:13 PM, Yingchao Lu <yingchao.lu at rice.edu> wrote:
>
>> Hi All,
>>
>>
>>
>> I want to do a time query over multiply hdf5 files as following:
>>
>>
>>
>> import yt
>>
>> from glob import glob
>>
>>
>>
>> fns = glob(‘flash_hdf5_plf_*’)
>>
>> data = []
>>
>>
>>
>> for fn in fns:
>>
>> ds = yt.load(fn)
>>
>> pt = ds.point([0,0,0])
>>
>> data.append([ds.parameters[‘time’], pt.mean(‘tele’)])
>>
>> ds.close()
>>
>>
>>
>> print data
>>
>>
>>
>> At the beginning, it takes about 1s to process each file. But later, it
>> becomes slower and slower. Finally, 100 files cost 10 minutes, much large
>> than 1s*100. I guess maybe the database is not fully closed.
>>
>
> Doing "del ds" should completely close and free the dataset object
> (although python's garbage collector should be doing that each iteration of
> your for loop).
>
> There are some operations in your script (in particular "pt.mean('tele')")
> that might be slower later in the time series, especially if the number of
> cells in the simulation increases with time.
>
>

To make this a bit more concrete, I went ahead and made a test script that
uses some of the public data on yt-project.org/data. You can find it here:

http://paste.yt-project.org/show/7018/

This particular time series has 70 outputs, so not as many as yours.

The output I get from running this script is here:

http://paste.yt-project.org/show/7019/

Each loop iteration takes about a second and I don't see any appreciable
increase as we get later in the time series.

When you run that script on your time series (replacing pt['temp'] with
pt['tele']), do you get different answers? If so can you share them on the
yt pastebin? Can you also share which yt version you're using?

If you can also share the dataset that's triggering this behavior, that
would also be helpful. If it's ok to publicly post it, you can share your
dataset using the yt curldrop:

https://docs.hub.yt/services.html#curldrop


> When I try ds.field_info() after ds.close(), it still displays the field
>> information. Does anyone have the same problem?
>>
>>
>>
>>
>>
>> Yingchao
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20170207/743ade85/attachment.html>