[yt-users] How to fully close a database

Nathan Goldbaum nathan12343 at gmail.com
Tue Feb 7 15:58:38 PST 2017


On Tue, Feb 7, 2017 at 4:55 PM, Yingchao Lu <yingchao.lu at rice.edu> wrote:

> Hi Nathan,
>
>
>
> I use yt 3.3.2. The number of cells in the simulation is fixed. The slow
> down just happens randomly, may be due to other r/w processes on the same
> cluster. I tried to make a big tarball when running yt script, it did slow
> down. Otherwise, the time seems normal.
>

That makse sense. These sort of i/o slowdowns are common on clusters with
large shared filesystems. Parallel filesystems (e.g. lustre) can make this
worse.


>
>
> BTW, in the output from your script, it does not have something like this:
>
> yt : [INFO     ] 2017-02-07 16:46:25,010 integer runtime parameter
> checkpointfilenumber overwrites a simulation scalar of the same name
>
> yt : [INFO     ] 2017-02-07 16:46:25,011 integer runtime parameter
> plotfilenumber overwrites a simulation scalar of the same name
>
> yt : [INFO     ] 2017-02-07 16:46:25,029 Parameters:
> current_time              = 2.40030489143e-09
>
> yt : [INFO     ] 2017-02-07 16:46:25,029 Parameters:
> domain_dimensions         = [16 16 32]
>
> yt : [INFO     ] 2017-02-07 16:46:25,029 Parameters:
> domain_left_edge          = [-0.15 -0.15 -0.05]
>
> yt : [INFO     ] 2017-02-07 16:46:25,030 Parameters:
> domain_right_edge         = [ 0.15  0.15  0.55]
>
> yt : [INFO     ] 2017-02-07 16:46:25,030 Parameters:
> cosmological_simulation   = 0.0
>
>
>
> How can I turn the message off?
>

These are logging messages. You can turn them off by adjusting the default
log level in your configuration file or by setting the log level at the top
of the script. This is covered in the docs here:

http://yt-project.org/doc/faq/index.html#how-can-i-change-yt-s-log-level

It's not mentioned there, but the configuration system also lets you
disable logging entirely via the suppressStreamLogging option:

http://yt-project.org/doc/reference/configuration.html#avail
able-configuration-options


>
>
> Thanks,
>
> Yingchao
>
>
>
> *From:* Nathan Goldbaum [mailto:nathan12343 at gmail.com]
> *Sent:* Tuesday, February 7, 2017 1:56 PM
> *To:* yingchao.lu at rice.edu; Discussion of the yt analysis package <
> yt-users at lists.spacepope.org>
> *Subject:* Re: [yt-users] How to fully close a database
>
>
>
>
>
>
>
> On Tue, Feb 7, 2017 at 1:42 PM, Nathan Goldbaum <nathan12343 at gmail.com>
> wrote:
>
>
>
>
>
> On Tue, Feb 7, 2017 at 1:13 PM, Yingchao Lu <yingchao.lu at rice.edu> wrote:
>
> Hi All,
>
>
>
> I want to do a time query over multiply hdf5 files as following:
>
>
>
> import yt
>
> from glob import glob
>
>
>
> fns = glob(‘flash_hdf5_plf_*’)
>
> data = []
>
>
>
> for fn in fns:
>
> ds = yt.load(fn)
>
> pt = ds.point([0,0,0])
>
> data.append([ds.parameters[‘time’], pt.mean(‘tele’)])
>
> ds.close()
>
>
>
> print data
>
>
>
> At the beginning, it takes about 1s to process each file. But later, it
> becomes slower and slower. Finally, 100 files cost 10 minutes, much large
> than 1s*100. I guess maybe the database is not fully closed.
>
>
>
> Doing "del ds" should completely close and free the dataset object
> (although python's garbage collector should be doing that each iteration of
> your for loop).
>
>
>
> There are some operations in your script (in particular "pt.mean('tele')")
> that might be slower later in the time series, especially if the number of
> cells in the simulation increases with time.
>
>
>
>
>
> To make this a bit more concrete, I went ahead and made a test script that
> uses some of the public data on yt-project.org/data. You can find it here:
>
>
>
> http://paste.yt-project.org/show/7018/
>
>
>
> This particular time series has 70 outputs, so not as many as yours.
>
>
>
> The output I get from running this script is here:
>
>
>
> http://paste.yt-project.org/show/7019/
>
>
>
> Each loop iteration takes about a second and I don't see any appreciable
> increase as we get later in the time series.
>
>
>
> When you run that script on your time series (replacing pt['temp'] with
> pt['tele']), do you get different answers? If so can you share them on the
> yt pastebin? Can you also share which yt version you're using?
>
>
>
> If you can also share the dataset that's triggering this behavior, that
> would also be helpful. If it's ok to publicly post it, you can share your
> dataset using the yt curldrop:
>
>
>
> https://docs.hub.yt/services.html#curldrop
>
>
>
> When I try ds.field_info() after ds.close(), it still displays the field
> information. Does anyone have the same problem?
>
>
>
>
>
> Yingchao
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20170207/c0cf1f2f/attachment-0001.htm>


More information about the yt-users mailing list