[yt-users] yt unpacking error on Pleiades

Nathan Goldbaum nathan12343 at gmail.com
Tue Jul 25 13:15:04 PDT 2017


Are you sure that the files are the same on your laptop and on Pleiades?
Are you sure that you're running the same version of yt on your laptop and
pleiades?

A way to check the former would be to make a tarball containing the dataset
and then find the tarball's SHA hash on your laptop and on Pleiades, e.g.:

# on linux
$ sha256sum my_data.tar

# on MacOS
$ shasum -a 256 my_data.tar

If those commands don't print the same answer on your laptop and on
Pleiades, that indicates that the files are not identical and perhaps there
has been some corruption.  That might be the source of the issue.

You can check to make sure the yt versions are the same on your laptop and
on Pleiades by running "yt version" on the bash command line.

What's happening in the error you're running that yt is trying to parse the
octree hierarchy in your data by reading some bytes from the data file.
It's dying because it's getting back a piece of data that's not the shape
it's expecting based on what we know about the ART data format. It's very
odd that this works on your laptop using the exact same data file, because
yt is making the same assumptions about the structure of the data file on
Pleiades and your laptop.

Finally, if none of the diagnostic suggestions I made above help, it would
help to make further progress on this issue if you can share the data file
that's triggering the problem so that one of us can reproduce the issue and
debug locally, although I totally understand if there are possible concerns
about sharing data.

-Nathan

On Tue, Jul 25, 2017 at 3:05 PM, Clayton Strawn <cjstrawn at ucsc.edu> wrote:

> Hello,
>
>
> I have a request for help.
>
> I am trying to run some analysis code remotely through ssh in yt on NASA
> Pleiades. I have been able to install yt and trident and am running in a
> virtualenv, but something is having a very deeply buried error that I don't
> understand and fatally kills any task I send to Pleiades.
>
> The same code works fine on my home laptop with the same input files, so I
> assume the problem is something to do with the yt distribution on Pleiades,
> but I used "pip install -U yt" so it should be up-to-date. The error code
> comes from the fortran_utils file, and it says "unpack requires a string
> argument of length 4"
>
>
> (myenv) cstrawn at pfe27:~/quasarlines/quasarlines> python quasar_scan.py
> VELA_v2_08 10MpcBox_csf512_a0.330.d "[H I,H II,O IV,O V,O VI,Si II,Si
> III,Si IV,C I,C II,C III,C IV,Mg II,Mg III,Mg IV]"
> yt : [INFO     ] 2017-07-25 12:11:26,454 discovered
> particle_data:/home1/cstrawn/quasarlines/quasarlines/PMcrs0a0.330.DAT
> yt : [INFO     ] 2017-07-25 12:11:26,454 discovered particle_header:/home1/
> cstrawn/quasarlines/quasarlines/PMcrda0.330.DAT
> yt : [INFO     ] 2017-07-25 12:11:26,462 Using root level of 14
> yt : [INFO     ] 2017-07-25 12:11:26,463 Discovered 7 species of particles
> yt : [INFO     ] 2017-07-25 12:11:26,463 Particle populations:  28016640
> 4263936    752640    123904     19552   2089406    85309
> yt : [INFO     ] 2017-07-25 12:11:26,658 Max level is 12
> yt : [INFO     ] 2017-07-25 12:11:26,682 Parameters: current_time
>      = 3.350407555 Gyr
> yt : [INFO     ] 2017-07-25 12:11:26,682 Parameters: domain_dimensions
>     = [128 128 128]
> yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters: domain_left_edge
>      = [ 0.  0.  0.]
> yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters: domain_right_edge
>     = [ 1.  1.  1.]
> yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters:
> cosmological_simulation   = True
> yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters: current_redshift
>      = 2.02694081565
> yt : [INFO     ] 2017-07-25 12:11:26,683 Parameters: omega_lambda
>      = 0.730000019073
> yt : [INFO     ] 2017-07-25 12:11:26,684 Parameters: omega_matter
>      = 0.270000010729
> yt : [INFO     ] 2017-07-25 12:11:26,684 Parameters: hubble_constant
>     = 0.699999988079
> 18426 LOSs to scan.
> Is this an appropriate number? (0 for no)
> 1
> 1/18426
> Traceback (most recent call last):
>   File "quasar_scan.py", line 375, in <module>
>     q.get_coldens(save = 10)
>   File "quasar_scan.py", line 148, in get_coldens
>     ftype='gas')
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py",
> line 239, in make_simple_ray
>     particle_type = _determine_particle_type_from_ftype(ds, ftype)
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py",
> line 590, in _determine_particle_type_from_ftype
>     field_list_arr = np.asarray(ds.derived_field_list)
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py",
> line 164, in ireq
>     self.index
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py",
> line 419, in index
>     self, dataset_type=self.dataset_type)
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
> line 65, in __init__
>     super(ARTIndex, self).__init__(ds, dataset_type)
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/geometry_handler.py",
> line 50, in __init__
>     self._setup_geometry()
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/oct_geometry_handler.py",
> line 25, in _setup_geometry
>     self._initialize_oct_handler()
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
> line 91, in _initialize_oct_handler
>     self.domains]
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
> line 761, in level_count
>     self.level_offsets
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
> line 782, in level_offsets
>     self.ds.min_level, self.ds.max_level)
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
> line 820, in _count_art_octs
>     ns = fpu.peek_record_size(f, endian='>')
>   File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/utilities/fortran_utils.py",
> line 250, in peek_record_size
>     s = struct.unpack('>i', f.read(struct.calcsize('>i')))
> struct.error: unpack requires a string argument of length 4
>
>
> Please let me know if this is a known problem or if you have any idea what
> I should do to avoid running into it. I don't really understand fortran
> enough to know what is even supposed to be happening in fortran_utils.
>
> Thanks so much,
> Clayton Strawn
>
>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20170725/a31691d0/attachment-0001.htm>


More information about the yt-users mailing list