[yt-users] yt unpacking error on Pleiades
Clayton Strawn
cjstrawn at ucsc.edu
Tue Jul 25 15:10:18 PDT 2017
Well, I guess it wasn't exactly the same file. On my computer I have other
galaxy snapshots from the same code, in the same format, which I assumed
would be practically identical except for different numbers
But when I tried porting it over to my computer it had the same error, and
another file on Pleiades of a different snapshot worked fine. Apparently it
was an error with the data storage/retrieval, and not with yt. So now it's
working.
Thanks so much for your quick response though!
Clayton
On Tue, Jul 25, 2017 at 1:15 PM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:
> Are you sure that the files are the same on your laptop and on Pleiades?
> Are you sure that you're running the same version of yt on your laptop and
> pleiades?
>
> A way to check the former would be to make a tarball containing the
> dataset and then find the tarball's SHA hash on your laptop and on
> Pleiades, e.g.:
>
> # on linux
> $ sha256sum my_data.tar
>
> # on MacOS
> $ shasum -a 256 my_data.tar
>
> If those commands don't print the same answer on your laptop and on
> Pleiades, that indicates that the files are not identical and perhaps there
> has been some corruption. That might be the source of the issue.
>
> You can check to make sure the yt versions are the same on your laptop and
> on Pleiades by running "yt version" on the bash command line.
>
> What's happening in the error you're running that yt is trying to parse
> the octree hierarchy in your data by reading some bytes from the data
> file. It's dying because it's getting back a piece of data that's not the
> shape it's expecting based on what we know about the ART data format. It's
> very odd that this works on your laptop using the exact same data file,
> because yt is making the same assumptions about the structure of the data
> file on Pleiades and your laptop.
>
> Finally, if none of the diagnostic suggestions I made above help, it would
> help to make further progress on this issue if you can share the data file
> that's triggering the problem so that one of us can reproduce the issue and
> debug locally, although I totally understand if there are possible concerns
> about sharing data.
>
> -Nathan
>
> On Tue, Jul 25, 2017 at 3:05 PM, Clayton Strawn <cjstrawn at ucsc.edu> wrote:
>
>> Hello,
>>
>>
>> I have a request for help.
>>
>> I am trying to run some analysis code remotely through ssh in yt on NASA
>> Pleiades. I have been able to install yt and trident and am running in a
>> virtualenv, but something is having a very deeply buried error that I don't
>> understand and fatally kills any task I send to Pleiades.
>>
>> The same code works fine on my home laptop with the same input files, so
>> I assume the problem is something to do with the yt distribution on
>> Pleiades, but I used "pip install -U yt" so it should be up-to-date. The
>> error code comes from the fortran_utils file, and it says "unpack
>> requires a string argument of length 4"
>>
>>
>> (myenv) cstrawn at pfe27:~/quasarlines/quasarlines> python quasar_scan.py
>> VELA_v2_08 10MpcBox_csf512_a0.330.d "[H I,H II,O IV,O V,O VI,Si II,Si
>> III,Si IV,C I,C II,C III,C IV,Mg II,Mg III,Mg IV]"
>> yt : [INFO ] 2017-07-25 12:11:26,454 discovered
>> particle_data:/home1/cstrawn/quasarlines/quasarlines/PMcrs0a0.330.DAT
>> yt : [INFO ] 2017-07-25 12:11:26,454 discovered
>> particle_header:/home1/cstrawn/quasarlines/quasarlines/PMcrda0.330.DAT
>> yt : [INFO ] 2017-07-25 12:11:26,462 Using root level of 14
>> yt : [INFO ] 2017-07-25 12:11:26,463 Discovered 7 species of particles
>> yt : [INFO ] 2017-07-25 12:11:26,463 Particle populations: 28016640
>> 4263936 752640 123904 19552 2089406 85309
>> yt : [INFO ] 2017-07-25 12:11:26,658 Max level is 12
>> yt : [INFO ] 2017-07-25 12:11:26,682 Parameters: current_time
>> = 3.350407555 Gyr
>> yt : [INFO ] 2017-07-25 12:11:26,682 Parameters: domain_dimensions
>> = [128 128 128]
>> yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: domain_left_edge
>> = [ 0. 0. 0.]
>> yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: domain_right_edge
>> = [ 1. 1. 1.]
>> yt : [INFO ] 2017-07-25 12:11:26,683 Parameters:
>> cosmological_simulation = True
>> yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: current_redshift
>> = 2.02694081565
>> yt : [INFO ] 2017-07-25 12:11:26,683 Parameters: omega_lambda
>> = 0.730000019073
>> yt : [INFO ] 2017-07-25 12:11:26,684 Parameters: omega_matter
>> = 0.270000010729
>> yt : [INFO ] 2017-07-25 12:11:26,684 Parameters: hubble_constant
>> = 0.699999988079
>> 18426 LOSs to scan.
>> Is this an appropriate number? (0 for no)
>> 1
>> 1/18426
>> Traceback (most recent call last):
>> File "quasar_scan.py", line 375, in <module>
>> q.get_coldens(save = 10)
>> File "quasar_scan.py", line 148, in get_coldens
>> ftype='gas')
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py",
>> line 239, in make_simple_ray
>> particle_type = _determine_particle_type_from_ftype(ds, ftype)
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/trident/ray_generator.py",
>> line 590, in _determine_particle_type_from_ftype
>> field_list_arr = np.asarray(ds.derived_field_list)
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py",
>> line 164, in ireq
>> self.index
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/data_objects/static_output.py",
>> line 419, in index
>> self, dataset_type=self.dataset_type)
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
>> line 65, in __init__
>> super(ARTIndex, self).__init__(ds, dataset_type)
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/geometry_handler.py",
>> line 50, in __init__
>> self._setup_geometry()
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/geometry/oct_geometry_handler.py",
>> line 25, in _setup_geometry
>> self._initialize_oct_handler()
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
>> line 91, in _initialize_oct_handler
>> self.domains]
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
>> line 761, in level_count
>> self.level_offsets
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
>> line 782, in level_offsets
>> self.ds.min_level, self.ds.max_level)
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/frontends/art/data_structures.py",
>> line 820, in _count_art_octs
>> ns = fpu.peek_record_size(f, endian='>')
>> File "/u/cstrawn/myenv/lib/python2.7/site-packages/yt/utilities/fortran_utils.py",
>> line 250, in peek_record_size
>> s = struct.unpack('>i', f.read(struct.calcsize('>i')))
>> struct.error: unpack requires a string argument of length 4
>>
>>
>> Please let me know if this is a known problem or if you have any idea
>> what I should do to avoid running into it. I don't really understand
>> fortran enough to know what is even supposed to be happening in
>> fortran_utils.
>>
>> Thanks so much,
>> Clayton Strawn
>>
>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20170725/36e397e4/attachment.htm>
More information about the yt-users
mailing list