[yt-users] Hop in parallel?
Michael Kuhlen
mqk at astro.berkeley.edu
Fri Feb 24 15:36:29 PST 2012
Sorry to be the bearer of bad news again, but ParallelHF failed for me.
The installation of Forthon, etc. went smoothly. Only took 5 minutes.
Note that in addition to replacing HaloFinder with
parallelHF you also have to add "from
yt.analysis_modules.halo_finding.api import *" to the top of the
script.
The error I encounter seems to be related to the large grid count that
was also giving me problems yesterday. Here's the traceback.
File "test.py", line 6, in <module>
halo_list = parallelHF(pf)
File "/home/mqk/local/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/analysis_modules/halo_finding/halo_objects.py",
line 2047, in __init__
total_mass =
self.comm.mpi_allreduce((self._data_source["ParticleMassMsun"].astype('float64')).sum(),
File "/home/mqk/local/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
line 321, in __getitem__
self.get_data(key)
File "/home/mqk/local/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/data_objects/data_containers.py",
line 2423, in get_data
self.particles.get_data(field)
File "/home/mqk/local/lib/python2.7/site-packages/yt-2.4dev-py2.7-linux-x86_64.egg/yt/data_objects/particle_io.py",
line 99, in get_data
count=len(grid_list), dtype='float64'))
ValueError: iterator too short
Again it's the total_mass calculation that is causing the problem.
Note that this yt install includes the changes that Matt brought in
yesterday to fix the memory consumption.
When I specify total_mass manually, everything works and I get the
correct number of halos in the final catalog. The number of halos and
their properties are not exactly identical to serial Hop, but it's
close enough. I got a speed-up of about a factor of 3, when running
parallelHF on 8 cores vs. HaloFinder on 1 core.
Mike
On Fri, Feb 24, 2012 at 2:37 PM, Michael Kuhlen <mqk at astro.berkeley.edu> wrote:
> No, I haven't yet. From the documentation ParallelHop seems to require
> installing Forthon, and I was lazy. ;) I'll give that a shot. Thanks
> for the suggestion.
>
> Mike
>
> On Fri, Feb 24, 2012 at 2:31 PM, Britton Smith <brittonsmith at gmail.com> wrote:
>> Hi Mike,
>>
>> Perhaps I missed something from the previous discussion. I've been locked
>> away working on proposals all week. Out of curiosity, have you tried using
>> ParallelHop, which is Stephen's specially designed parallel Hop. It is
>> entirely distinct from simply running hop in parallel. Stephen, please
>> correct me if I'm wrong.
>>
>> You should be able to use that simply by replacing HaloFinder with
>> parallelHF in your script.
>>
>> Britton
>>
>> On Fri, Feb 24, 2012 at 5:23 PM, Stephen Skory <s at skory.us> wrote:
>>>
>>> Hi Mike,
>>>
>>> > Is running Hop in parallel currently broken? I get way fewer halos in
>>> > my catalog when I run Hop with 8 processors than with 1. Am I doing
>>> > something wrong?
>>>
>>> Why do you always got to start trouble? I'm getting this too. I'll dig
>>> into this.
>>>
>>> --
>>> Stephen Skory
>>> s at skory.us
>>> http://stephenskory.com/
>>> 510.621.3687 (google voice)
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>
>
>
> --
> *********************************************************************
> * *
> * Dr. Michael Kuhlen Theoretical Astrophysics Center *
> * email: mqk at astro.berkeley.edu UC Berkeley *
> * cell phone: (831) 588-1468 B-116 Hearst Field Annex # 3411 *
> * skype username: mikekuhlen Berkeley, CA 94720 *
> * *
> *********************************************************************
--
*********************************************************************
* *
* Dr. Michael Kuhlen Theoretical Astrophysics Center *
* email: mqk at astro.berkeley.edu UC Berkeley *
* cell phone: (831) 588-1468 B-116 Hearst Field Annex # 3411 *
* skype username: mikekuhlen Berkeley, CA 94720 *
* *
*********************************************************************
More information about the yt-users
mailing list