[yt-users] Having trouble running yt on kraken

[Matthew Turk]

Hi Yuan,

On Sat, Jul 28, 2012 at 8:22 PM, Yuan Li <bear0980 at gmail.com> wrote:
> Hi Matt,
>
> I was making thin projections. Each output dataset (DD****) is about 14G.
> What are "multiple root level tiles"? I have 256^3 root grids...

Hm, you should be able to project no problem, even in serial on a
laptop.  I think the problem might be in an option that we recently
changed, which was over-allocated memory for operations like these.
We're putting out a release very soon (over on yt-dev we're even
talking about drafting the release announcement!) which would be a
good time to upgrade.  The issue that was happening in the past was
that all grids for a given data source were being loaded at
initializatio nof the projection; plus, recent fixes to how the
projection was reduced in parallel should reduce peak memory usage but
keep it also steady at that peak.

This also sounds like a good use case for a recipe in the cookbook
that's going out with the documentation in the new release.  Would you
mind sending me a description or script (off-list) that I could
convert to use the new stuff in the next release, which should help
out with memory?  We've done a lot to improve analysis of many outputs
at once, and in particularly for doing this in parallel.  I'd love to
add this as a worked example.

Here are a few using the new features in 2.4 (which are all in the
development branch -- everything's developed in the open) that might
show how we might be able to reduce memory overhead and speed up
overall computation time.

http://yt-project.org/docs/2.4/cookbook/calculating_information.html#time-series-analysis
http://yt-project.org/docs/2.4/advanced/parallel_computation.html#an-advanced-worked-example

>
> I just submitted another job asking for more nodes. I will let you know how
> that goes (in two days).

Cool.  :)

-Matt

>
> Thanks!
> Yuan
>
>
> On Sat, Jul 28, 2012 at 7:24 PM, Matthew Turk <matthewturk at gmail.com> wrote:
>>
>> Hi Yuan,
>>
>> On Sat, Jul 28, 2012 at 6:41 PM, Yuan Li <bear0980 at gmail.com> wrote:
>> > Hi Britton,
>> >
>> > Thanks for your suggestion! Now the problem has been semi-solved.
>> >
>> > I have checked my script and I have this "module load yt/dev/", so I
>> > guess I
>> > am using yt/dev, which is what I use on other machines.
>> >
>> > There is indeed a problem with "slice". I had all my functions defined
>> > inside one file called profile.py (my bad: I started with profiles and
>> > kept
>> > adding things without changing the name of it). After I changed that
>> > file to
>> > "yuan.py", the error I got was:
>> > ......
>> >
>> > Traceback (most recent call last):
>> >   File "/var/spool/alps/2362242/try1.py", line 3, in <module>
>> >     import yuan as yuan
>> > ImportError: No module named yuan
>> > ......
>> >
>> > So Kraken did not see my file. It did not complain before because there
>> > was
>> > a built-in function in python called profile.
>> >
>> > Now after I added this to my script:
>> > import sys
>> > sys.path.append('/lustre/scratch/yuan/Thermal09/')
>> >
>> > Kraken could finally import my functions!
>> >
>> >
>> > The reason why I said that it was only semi-solved is because now I am
>> > getting "MemoryError". I will try to use more nodes, but it may never
>> > work
>> > :).
>>
>> What operation are you doing, and how big is your dataset?  And, does
>> your data have multiple root level tiles?
>>
>> -Matt
>>
>> >
>> > How many nodes do you use for each job, Britton?
>> >
>> > Thanks!
>> > Yuan
>> >
>> >
>> > On Thu, Jul 26, 2012 at 6:39 PM, Britton Smith <brittonsmith at gmail.com>
>> > wrote:
>> >>
>> >> Hi Yuan,
>> >>
>> >> This could be a result of using different version of yt on the two
>> >> different platforms.  Can you check and see which version of yt you're
>> >> using
>> >> on kraken?  I believe there are modules for versions 2.2, 2.3, and the
>> >> development branch on kraken.
>> >>
>> >> Also, the variable "slice" is already used in python for something
>> >> else.
>> >> If slice is apart of your script, you may want to rename that to
>> >> something
>> >> else.
>> >>
>> >> If your script only works on the most recent version of yt, you'll
>> >> probably want to try to use the yt/dev module on Kraken.
>> >>
>> >> Britton
>> >>
>> >> On Wed, Jul 25, 2012 at 5:18 PM, Yuan Li <bear0980 at gmail.com> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> I have been having trouble running yt on kraken. I am using the same
>> >>> script that I use on trestles (where it sort of works). The issue I
>> >>> have on
>> >>> kraken is that when I submit a non-interactive job, I get this error:
>> >>> + aprun -n 12 ./try1.py --parallel
>> >>> Traceback (most recent call last):
>> >>>   File "/var/spool/alps/2310447/try1.py", line 3, in <module>
>> >>>     slice(imin=12, imax=47, width=20)
>> >>> TypeError: slice() does not take keyword arguments
>> >>>
>> >>>
>> >>> If I run an interactive job (./try1.py --parallel), I get this error:
>> >>> yuan at krakenpf2:~/scratch/Thermal09> Assertion failed in file
>> >>>
>> >>> /ptmp/ulib/mpt/nightly/5.2/051011/mpich2/src/mpid/cray/src/adi/mpid_init.c
>> >>> at line 115: 0
>> >>> aborting job:
>> >>> (null)
>> >>>
>> >>>
>> >>> I can run a non-parallel interactive job (./try1.py) without any
>> >>> problem.
>> >>>
>> >>> My script looks like this:
>> >>> #!/lustre/scratch/proj/sw/yt/dev/bin/python
>> >>> from profile import *
>> >>> slice(imin=41, imax=47, width=20)
>> >>>
>> >>>
>> >>> Thanks!
>> >>> Yuan
>> >>>
>> >>> _______________________________________________
>> >>> yt-users mailing list
>> >>> yt-users at lists.spacepope.org
>> >>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> yt-users mailing list
>> >> yt-users at lists.spacepope.org
>> >> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >>
>> >
>> >
>> > _______________________________________________
>> > yt-users mailing list
>> > yt-users at lists.spacepope.org
>> > http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>> >
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>