<p dir="ltr"><br>
On Apr 11, 2014 7:36 PM, "B.W. Keller" <<a href="mailto:kellerbw@mcmaster.ca">kellerbw@mcmaster.ca</a>> wrote:<br>
><br>
> Alright everyone, I've fixed the problem. It looks like when we added support for pkdGrav's double precision positions, it was implemented in a rather slow way, the whole file needed to be loaded and processed (more than once). I've implemented all the filetype checking in a more efficient way. It now only reads ~30 bytes to check if a file is valid. Here's the PR:<br>
><br>
> <a href="https://bitbucket.org/yt_analysis/yt/pull-request/811/quickly-validate-tipsy-files/diff">https://bitbucket.org/yt_analysis/yt/pull-request/811/quickly-validate-tipsy-files/diff</a><br>
><br>
> Ben </p>
<p dir="ltr">Well, my face is red. Thanks to Ben for covering for me, but sounds like this was my fault! Sorry everyone.</p>
<p dir="ltr">Relatedly, answer tests are just about back online.</p>
<p dir="ltr">Matt</p>
<p dir="ltr">><br>
><br>
> On Fri, Apr 11, 2014 at 2:19 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
>><br>
>> I have implemented the suggested patch in this commit: <a href="https://bitbucket.org/yt_analysis/yt/commits/cc6b873a4471e049b6fef33146e92615c72f1243">https://bitbucket.org/yt_analysis/yt/commits/cc6b873a4471e049b6fef33146e92615c72f1243</a><br>
>><br>
>><br>
>> On Fri, Apr 11, 2014 at 10:58 AM, B.W. Keller <<a href="mailto:kellerbw@mcmaster.ca">kellerbw@mcmaster.ca</a>> wrote:<br>
>>><br>
>>> Yep, that's the quickest fix. I'll send a fix up later today. Sorry about this folks.<br>
>>><br>
>>><br>
>>> On Fri, Apr 11, 2014 at 1:18 PM, Sam Skillman <<a href="mailto:samskillman@gmail.com">samskillman@gmail.com</a>> wrote:<br>
>>>><br>
>>>> +1 to John's implementation. In the interim, tipsy can then still be loaded explicitly.<br>
>>>><br>
>>>><br>
>>>> On Fri, Apr 11, 2014 at 10:13 AM, John ZuHone <<a href="mailto:jzuhone@gmail.com">jzuhone@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>> Sorry, being a bit more descriptive... by that I mean:<br>
>>>>><br>
>>>>> @classmethod<br>
>>>>> def _is_valid(self, *args, **kwargs):<br>
>>>>> return False<br>
>>>>> ...<br>
>>>>><br>
>>>>> Which is a bit hacky, but the least invasive.<br>
>>>>><br>
>>>>> On Apr 11, 2014, at 1:07 PM, John ZuHone <<a href="mailto:jzuhone@gmail.com">jzuhone@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>>> +1, except it seems like the easiest thing to do is to just make sure that Tipsy's _is_valid always returns False for now.<br>
>>>>>><br>
>>>>>> On Apr 11, 2014, at 12:41 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br>
>>>>>><br>
>>>>>>> Hi all,<br>
>>>>>>><br>
>>>>>>> While Ben works on his fix, would anyone object to temporarily reverting the merged pull request that triggered this behavior?<br>
>>>>>>><br>
>>>>>>> I worry about FLASH users who do not read yt-dev. I guess it *is* the bleeding edge, experimental version so bugs should be expected but still, reverting seems like an easy temporary fix that takes some pressure off Ben to quickly develop a true fix to the underlying issue.<br>
>>>>>>><br>
>>>>>>> Nathan<br>
>>>>>>><br>
>>>>>>> On Thursday, April 10, 2014, B.W. Keller <<a href="mailto:kellerbw@mcmaster.ca">kellerbw@mcmaster.ca</a>> wrote:<br>
>>>>>>>><br>
>>>>>>>> Good idea Mike! I'll do that too.<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> On Thu, Apr 10, 2014 at 7:39 PM, Mike Warren <<a href="mailto:mswarren@gmail.com">mswarren@gmail.com</a>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>> You could detect that it is not a tipsy file quickly, by using the<br>
>>>>>>>>> constraint in the header that nbodies=nsph+ndark+nstar and ndim is<br>
>>>>>>>>> presumably 1,2 or 3.<br>
>>>>>>>>><br>
>>>>>>>>> struct tipsy_dump {<br>
>>>>>>>>> double time;<br>
>>>>>>>>> int nbodies;<br>
>>>>>>>>> int ndim;<br>
>>>>>>>>> int nsph;<br>
>>>>>>>>> int ndark;<br>
>>>>>>>>> int nstar;<br>
>>>>>>>>> };<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> On Thu, Apr 10, 2014 at 5:32 PM, B.W. Keller <<a href="mailto:kellerbw@mcmaster.ca">kellerbw@mcmaster.ca</a>> wrote:<br>
>>>>>>>>> > Oh my. Sorry that I have introduced this, unfortunately there is no way to<br>
>>>>>>>>> > detect tipsy files other than actually reading the entire file from disk.<br>
>>>>>>>>> > Perhaps the way to fix this would be to drop the priority of tipsy datasets<br>
>>>>>>>>> > to the bottom, so that a valid FLASH dataset will be detected prior to the<br>
>>>>>>>>> > Tipsy check?<br>
>>>>>>>>> ><br>
>>>>>>>>> ><br>
>>>>>>>>> > On Thu, Apr 10, 2014 at 7:15 PM, Nathan Goldbaum <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>><br>
>>>>>>>>> > wrote:<br>
>>>>>>>>> >><br>
>>>>>>>>> >> Hi all,<br>
>>>>>>>>> >><br>
>>>>>>>>> >> We just had a bug report from Aaron Smith at UT Austin. The symptom is<br>
>>>>>>>>> >> that the "load" comman was taking 30 seconds to complete on his FLASH<br>
>>>>>>>>> >> dataset, which should never happen for FLASH.<br>
>>>>>>>>> >><br>
>>>>>>>>> >> After asking him to profile the code, he produced the following profile:<br>
>>>>>>>>> >><br>
>>>>>>>>> >> <a href="http://ngoldbaum.net/yt-load/">http://ngoldbaum.net/yt-load/</a><br>
>>>>>>>>> >><br>
>>>>>>>>> >> It seems that the recent changes to the Tipsy frontend which allow it to<br>
>>>>>>>>> >> autodetect binary outputs have made it so in some cases non-tipsy data is<br>
>>>>>>>>> >> loaded off disk.<br>
>>>>>>>>> >><br>
>>>>>>>>> >> I'm not sure about the best way to handle this, which is why I'm writing<br>
>>>>>>>>> >> to the list rather than issuing a PR.<br>
>>>>>>>>> >><br>
>>>>>>>>> >> -Nathan<br>
>>>>>>>>> >><br>
>>>>>>>>> ><br>
>>>>>>>>> ><br>
>>>>>>>>> > _______________________________________________<br>
>>>>>>>>> > yt-dev mailing list<br>
>>>>>>>>> > <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>>>>>> > <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>>>>>> ><br>
>>>>>>>>> _______________________________________________<br>
>>>>>>>>> yt-dev mailing list<br>
>>>>>>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>>>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>> _______________________________________________<br>
>>>>>>> yt-dev mailing list<br>
>>>>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>>><br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>>> _______________________________________________<br>
>>>>> yt-dev mailing list<br>
>>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>>><br>
>>>><br>
>>>><br>
>>>> _______________________________________________<br>
>>>> yt-dev mailing list<br>
>>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> yt-dev mailing list<br>
>>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> yt-dev mailing list<br>
>> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
>> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> yt-dev mailing list<br>
> <a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
> <a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
><br>
</p>