[yt-users] HDF5 TypeError when writing LightRay

Mon Apr 24 15:17:57 PDT 2017

Nathan,

You are correct, the enzo simulation is from before 2012
PR2589 does indeed fix the issue I was having.
Thank you for your assistance.

Joe Tomlinson

On Mon, Apr 24, 2017 at 6:02 PM, Nathan Goldbaum <nathan12343 at gmail.com>
wrote:

> Hi Joseph,
>
> Thanks for sharing your dataset, that made it much easier to see what was
> going wrong. I've issued a pull request here that should fix things:
>
> https://bitbucket.org/yt_analysis/yt/pull-requests/2589
>
> The reason we hadn't seen this before is that you are working with a
> somewhat peculiar Enzo output that doesn't contain any unique identifier
> metadata. This metadata has been in the Enzo output format for several
> years now, so I guess this is a somewhat older dataset? In any case it was
> triggering a bug in yt that was causing ds.unique_identifier to get set to
> None, which then triggered another bug in the light ray saving machinery
> when it was passed such a strange unique identifier.
>
> I think I've fixed both issues. It would be great if you could give it a
> test. I'm happy to help out if you have questions about how to do that.
>
> -Nathan
>
> On Mon, Apr 24, 2017 at 4:20 PM, Joseph Tomlinson <jmtomlinson95 at gmail.com
> > wrote:
>
>> Nathan, Britton
>>
>> The dataset was made using a cosmological enzo simulation. I have
>> uploaded it here use.yt/upload/ad078d12 be aware the tarball is around
>> 5GB.
>>
>> Thanks,
>> Joe Tomlinson
>> Senior Undergraduate
>> Department of Physics, Drexel University
>>
>>
>> On Mon, Apr 24, 2017 at 5:15 PM, Britton Smith <brittonsmith at gmail.com>
>> wrote:
>>
>>> Hi Joseph, Nathan,
>>>
>>> The problem seems to be one of two things.  First, this is a light ray
>>> made by a single dataset, and so an hdf5 attribute that is normally
>>> expected to be a list of strings was a single single.  Second, this
>>> attribute, which records the unique_identifiers for the datasets used to
>>> make the light ray is "[None]", so whatever dataset this was is missing a
>>> unique_identifier attribute.  Fixing the first issue will probably take
>>> care of it, but it's probably worth making sure that this dataset type gets
>>> unique_attributes made as well.
>>>
>>> What type of dataset was used to make this light ray?
>>>
>>> Britton
>>>
>>> On Mon, Apr 24, 2017 at 4:02 PM, Nathan Goldbaum <nathan12343 at gmail.com>
>>> wrote:
>>>
>>>> Hmm, so this light ray dataset you've shared with me is missing a lot
>>>> of metadata. Is there any chance you can share the original dataset with me
>>>> so I can debug the light ray creation process as well? You should be able
>>>> to use the curldrop to upload the dataset as a tarball.
>>>>
>>>> On Mon, Apr 24, 2017 at 3:59 PM, Joseph Tomlinson <
>>>> jmtomlinson95 at gmail.com> wrote:
>>>>
>>>>> Nathan,
>>>>>
>>>>> To just generate the error run http://paste.yt-project.org/show/7130/
>>>>> after downloading the hdf5 file I uploaded.
>>>>> The generation of the hdf5 file requires the actual dataset I'm
>>>>> working with, but it is generated with http://paste.yt-project.org/sh
>>>>> ow/7131/.
>>>>>
>>>>> Thanks,
>>>>> Joe Tomlinson
>>>>> Senior Undergraduate
>>>>> Department of Physics, Drexel University
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 24, 2017 at 4:51 PM, Nathan Goldbaum <
>>>>> nathan12343 at gmail.com> wrote:
>>>>>
>>>>>> Hi Joseph,
>>>>>>
>>>>>> I'm unfamiliar with the LightRay interface - can you make a short
>>>>>> sample script that triggers the issue you're seeing? I want to make sure
>>>>>> we're on the same page.
>>>>>>
>>>>>> -Nathan
>>>>>>
>>>>>> On Mon, Apr 24, 2017 at 3:49 PM, Joseph Tomlinson <
>>>>>> jmtomlinson95 at gmail.com> wrote:
>>>>>>
>>>>>>> Nathan,
>>>>>>>
>>>>>>> I apologize if I was unclear. I am still having an issue even on the
>>>>>>> latest dev version. I was having a separate issue on the stable branch that
>>>>>>> I believe was the issue you linked me previously.
>>>>>>> The issue I am currently having is that whenever yt tries to load a
>>>>>>> LightRay from my dataset it throws the error
>>>>>>>
>>>>>>> AttributeError: 'str' object has no attribute 'astype'
>>>>>>>
>>>>>>> This occurs from simply loading the hdf5 file generated from a call
>>>>>>> to make_light_ray. It should be easily reproducible by attempting to load
>>>>>>> the file I uploaded at use.yt/upload/9fe32716.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Joe Tomlinson
>>>>>>> Senior Undergraduate
>>>>>>> Department of Physics, Drexel University
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 24, 2017 at 4:40 PM, Nathan Goldbaum <
>>>>>>> nathan12343 at gmail.com> wrote:
>>>>>>>
>>>>>>>> You can use the yt curldrop to share the dataset:
>>>>>>>>
>>>>>>>> https://docs.hub.yt/services.html#curldrop
>>>>>>>>
>>>>>>>> But if it's as you said and the issue is fixed already (e.g. you do
>>>>>>>> not have any issues when you build yt from latest version on the "yt"
>>>>>>>> branch in the mercurial repository) then there's no need to go further,
>>>>>>>> since the issue is fixed already. We will hopefully be able to do a yt 3.4
>>>>>>>> release soon after moving development to github, so that will get this fix
>>>>>>>> out there in a more easily shareable fashion.
>>>>>>>>
>>>>>>>> -Nathan
>>>>>>>>
>>>>>>>> On Mon, Apr 24, 2017 at 3:34 PM, Joseph Tomlinson <
>>>>>>>> jmtomlinson95 at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Nathan,
>>>>>>>>>
>>>>>>>>> That issue is very likely the problem that I was having with the
>>>>>>>>> stable version.
>>>>>>>>>
>>>>>>>>> I tried to recreate the issue I am having with the dev version but
>>>>>>>>> the issue does not occur for any of the public enzo datasets.
>>>>>>>>>
>>>>>>>>> The error occurs after the ray data has been saved and it reloads
>>>>>>>>> it . The saved file is able to be read in h5py and seems correct.
>>>>>>>>> A few layers down the error occurs in _parse_parameter_file() and
>>>>>>>>> I think this may be the difference between the data I was given and the
>>>>>>>>> public sets.
>>>>>>>>> I can't really see the difference between the two myself. I've
>>>>>>>>> uploaded the file here http://paste.yt-project.org/show/7129/ if
>>>>>>>>> it becomes relevant.
>>>>>>>>>
>>>>>>>>> The script to generate the error is extremely basic, I didn't
>>>>>>>>> upload it because its just two lines, making the LightRay then calling
>>>>>>>>> make_light_ray.
>>>>>>>>> The error should be reproducible by simply trying to load the
>>>>>>>>> resultant hdf5 file, is there somewhere you would prefer me to upload it?
>>>>>>>>> It's not too large, less than 1MB.
>>>>>>>>> Is there anything else I could provide to help diagnose the issue?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Joe Tomlinson
>>>>>>>>> Senior Undergraduate
>>>>>>>>> Department of Physics, Drexel University
>>>>>>>>>
>>>>>>>>> On Mon, Apr 24, 2017 at 2:09 PM, Nathan Goldbaum <
>>>>>>>>> nathan12343 at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Joseph,
>>>>>>>>>>
>>>>>>>>>> I believe you're hitting this issue:
>>>>>>>>>>
>>>>>>>>>> https://bitbucket.org/yt_analysis/yt/issues/1330
>>>>>>>>>>
>>>>>>>>>> Which was fixed here:
>>>>>>>>>>
>>>>>>>>>> https://bitbucket.org/yt_analysis/yt/pull-requests/2541
>>>>>>>>>>
>>>>>>>>>> We haven't had a stable release since that was fixed. If you're
>>>>>>>>>> running yt from a stable release, can you try installing the development
>>>>>>>>>> version from source? If you're using conda you can also use the nightly
>>>>>>>>>> binary builds:
>>>>>>>>>>
>>>>>>>>>> http://yt-project.org/docs/dev/installing.html#nightly-conda
>>>>>>>>>> -builds
>>>>>>>>>>
>>>>>>>>>> If you are running a recent development version then Britton's
>>>>>>>>>> fix might need to be eloborated upon. If so, can you create a
>>>>>>>>>> self-contained, runnable example script that triggers the issue you're
>>>>>>>>>> seeing (preferably making use of one of the public datasets on
>>>>>>>>>> yt-project.org/data)?
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> Nathan Goldbaum
>>>>>>>>>>
>>>>>>>>>> On Mon, Apr 24, 2017 at 1:04 PM, Joseph Tomlinson <
>>>>>>>>>> jmtomlinson95 at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to use the single dataset LightRay module but I am
>>>>>>>>>>> running into an issue.
>>>>>>>>>>> Whenever I use the data_filename parameter, which I believe is
>>>>>>>>>>> necessary for use with AbsorptionSpectrum, I receive the following error.
>>>>>>>>>>>
>>>>>>>>>>> TypeError: Object dtype dtype('O') has no native HDF5 equivalent
>>>>>>>>>>>
>>>>>>>>>>> This error does not occur when I run the cookbook example so I
>>>>>>>>>>> am a bit lost as to what to do.
>>>>>>>>>>> I am running the latest stable version on anaconda and have
>>>>>>>>>>> tried both h5py 2.7 and 2.6.
>>>>>>>>>>> The simulation is a cosmological Enzo simulation.
>>>>>>>>>>>
>>>>>>>>>>> I have also tried using the nightly dev build which changes the
>>>>>>>>>>> error from being on write to being on load with the following error.
>>>>>>>>>>>
>>>>>>>>>>> AttributeError: 'str' object has no attribute 'astype'
>>>>>>>>>>> from line 326 in _restore_light_ray_solution
>>>>>>>>>>> self.parameters[ksp3] = self.parameters[ksp3].astype(str)
>>>>>>>>>>>
>>>>>>>>>>> When using python 2.7 the error is the same but with 'unicode'
>>>>>>>>>>> instead of 'str'.
>>>>>>>>>>>
>>>>>>>>>>> Any help would be greatly appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Joe Tomlinson
>>>>>>>>>>> Senior Undergraduate
>>>>>>>>>>> Department of Physics, Drexel University
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> yt-users mailing list
>>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> yt-users mailing list
>>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> yt-users mailing list
>>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> yt-users mailing list
>>>>>>>> yt-users at lists.spacepope.org
>>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> yt-users mailing list
>>>>>>> yt-users at lists.spacepope.org
>>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> yt-users mailing list
>>>>>> yt-users at lists.spacepope.org
>>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> yt-users mailing list
>>>>> yt-users at lists.spacepope.org
>>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> yt-users mailing list
>>>> yt-users at lists.spacepope.org
>>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>>
>>>>
>>>
>>> _______________________________________________
>>> yt-users mailing list
>>> yt-users at lists.spacepope.org
>>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>>
>>>
>>
>> _______________________________________________
>> yt-users mailing list
>> yt-users at lists.spacepope.org
>> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>>
>>
>
> _______________________________________________
> yt-users mailing list
> yt-users at lists.spacepope.org
> http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.spacepope.org/pipermail/yt-users-spacepope.org/attachments/20170424/53fa36ed/attachment.html>