[yt-users] Parallel python/yt

[Nathan Goldbaum]

Hi Aaron,

Have you looked at the section of the yt documentation on parallelism?

http://yt-project.org/docs/dev/analyzing/parallel_computation.html

yt uses MPI-based parallelism (powered by the mpi4py package).  Using this
you can iterate over datasets in parallel using `parallel_objects` or
`TimeSeriesData.piter()`.

I'm not sure whether yt operations are safe when done inside a processing
pool like you're trying to do.  I think in principle it should work if all
of the processors are operating on different datasets but clearly something
is breaking somewhere...

In any case, I'd urge you to rewrite your script using yt's native
parallelism.

Please let us know if you have any further questions or concerns.

-Nathan


On Fri, Nov 15, 2013 at 10:43 AM, Wienkers, Aaron F. <wienkers1 at llnl.gov>wrote:

>  Ah yes, I meant to include this:
>     http://paste.yt-project.org/show/4059/
>
>  -Aaron
>
>   From: Sam Skillman <samskillman at gmail.com>
> Reply-To: Discussion of the yt analysis package <
> yt-users at lists.spacepope.org>
> Date: Thursday, November 14, 2013 9:44 PM
> To: Discussion of the yt analysis package <yt-users at lists.spacepope.org>
> Subject: Re: [yt-users] Parallel python/yt
>
>   Hi Aaron,
>
>  Could you post a bit more information, perhaps your script?  You can use
> http://paste.yt-project.org/ or do yt pastebin yourscript.py
>
>  Otherwise, my guess would be that you are attempting to open up the same
> hdf5 file with multiple threads, which is gaurded against by hdf5.
>
>  Best,
> Sam
>
>
> On Thu, Nov 14, 2013 at 7:04 PM, Wienkers, Aaron F. <wienkers1 at llnl.gov>wrote:
>
>>  Hi YT Users,
>>
>>    I've been trying to parallelize some of my analysis scripts using
>> multiprocessing pools, but after a few repetitions yt produces the error,
>> "Couldn't figure out output type for ./file_.hdf5" even though it works
>> fine up to that point. It doesn't seem to have anything to do with my hdf5
>> file since it works fine serially.
>>
>>  Just curious if anyone has run into a similar problem in the past.
>>
>>  Thanks,
>>   Aaron
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20131115/ba24beb4/attachment.htm>