<div dir="auto">Yes this is extremely useful. Thanks a lot for taking time to write me in detail about what went wrong. Just for info (in case its helpful) I would like to say that there's an open source module available called pygadgetreader which was able to successfully read all the available data in the Gadget 2 file I have shared.<div dir="auto"><br></div><div dir="auto">Cheers,<br><div data-smartmail="gmail_signature" dir="auto">Alankar</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Feb 23, 2017 12:42 AM, "Nathan Goldbaum" <<a href="mailto:nathan12343@gmail.com">nathan12343@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 22, 2017 at 12:17 PM, Alankar Dutta <span dir="ltr"><<a href="mailto:dutta.alankar@gmail.com" target="_blank">dutta.alankar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Hello,<br></div><div><br>The entire snapshot file is huge (around 1 TB) so I am attaching only the first part of it as of now via Google Drive. Hope this helps. I can share it via other means if necessary.<br></div></div></div></blockquote><div><br></div><div>Hmm, there seems to be an issue when yt tries to parse the header and validate that everything is consistent.</div><div><br></div><div>You can see the affected code block in yt here:</div><div><br></div><div><a href="https://bitbucket.org/yt_analysis/yt/src/abf5a8eff1b2d0cd776a41a33e5d3f3d25232ecc/yt/frontends/gadget/data_structures.py?at=yt&fileviewer=file-view-default#data_structures.py-340" target="_blank">https://bitbucket.org/yt_<wbr>analysis/yt/src/<wbr>abf5a8eff1b2d0cd776a41a33e5d3f<wbr>3d25232ecc/yt/frontends/<wbr>gadget/data_structures.py?at=<wbr>yt&fileviewer=file-view-<wbr>default#data_structures.py-340</a><br></div><div><br></div><div>This function is called (indirectly) by yt.load() to verify that a given file is *really* a Gadget binary file. When I follow along with the execution of that function for the file you sent me, I end up getting that np0 20736113 while np1 is 41472226.0 (i.e. exactly twice np0). The first integer, np0, is read in directly from the header of the binary file and corresponds to the number of particles in the dataset as written out to the header by Gadget. The second, np1. is the number of particles in the dataset inferred by reading in the size of the position block. The fact that inferring the particle count with the size of the position block ends up with exactly twice the number of particles as we expect probably indicates that there's a single/double precision issue. In fact, it seems that we expect each position entry to require 4 bytes (e.g. 32 bit, or single precision), so I infer that your file contains double precision positions (e.g. 8 bytes per particle).</div><div><br></div><div>So, all that to say, it looks like we would need to patch the Gadget frontend to support your output type, which seems to have double precision fields, which is a little bit different from the other Gadget binary outputs we've seen in the past. These sorts of issues with Gadget are unfortunately somewhat common due to the fragmentation in the Gadget ecosystem, with many research groups maintaining mutually incompatible versions of Gadget.</div><div><br></div><div>Note that if you look at page 32 in the Gadget user guide (<a href="https://wwwmpa.mpa-garching.mpg.de/gadget/users-guide.pdf" target="_blank">https://wwwmpa.mpa-garching.<wbr>mpg.de/gadget/users-guide.pdf</a>)<wbr>, this *is* a bit different from the output format documented there, which specifies single precision positions.</div><div><br></div><div>To add support for this output type we'd need to start with an example smallish (<5 GB) dataset in this format that we can add as a public test dataset on <a href="http://yt-project.org/data" target="_blank">yt-project.org/data</a>. Once that's available, we can patch the Gadget frontend to support this output type.</div><div><br></div><div>Finally, you mentioned that your dataset is pretty large (~1 TB). Unfortunately, yt will currently have trouble scaling to datasets that large. Right now yt will require substantial amounts of RAM to index datasets larger than about 1024^3 particles, since yt makes use of a global octree for indexing and managing I/O chunking. With a dataset so large, the octree index requires a substantial amount of RAM.</div><div><br></div><div>I am currently actively working on improving yt's scaling for large particle datasets. This is a major development effort that will likely be included in either the next major release of yt or the one after that. Unfortunately I think you will have lots of issues trying to get yt's current particle support to work well with as big of a dataset as you need to work with and you will likely need to wait until the development effort I'm working on is publicly available. I'd encourage you to sign up to the yt-dev mailing list if you want to hear more about this effort. I will be sharing a design document there describing the changes to yt that will be necessary to improve scaling for particle data in the next week or two.</div><div><br></div><div>I hope that's helpful,</div><div><br></div><div>Nathan Goldbaum</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div>Cheers,<br></div>Alankar<br><div class="gmail_chip gmail_drive_chip" style="width:396px;height:18px;max-height:18px;background-color:rgb(245,245,245);padding:5px;color:rgb(34,34,34);font-family:arial;font-style:normal;font-weight:bold;font-size:13px;border:1px solid rgb(221,221,221);line-height:1"><a href="https://drive.google.com/file/d/0B6IIQdUdRX9UN3dWRkdLWWxjNjg/view?usp=drive_web" style="display:inline-block;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;text-decoration:none;padding:1px 0px;border:medium none;width:100%" target="_blank"><img style="vertical-align:bottom;border:none" src="https://ssl.gstatic.com/docs/doclist/images/icon_10_generic_list.png"> <span dir="ltr" style="color:rgb(17,85,204);text-decoration:none;vertical-align:bottom">snapshot_068.0</span></a></div><br></div><div class="m_6449699038156499989gmail-HOEnZb"><div class="m_6449699038156499989gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 22, 2017 at 10:37 PM, Nathan Goldbaum <span dir="ltr"><<a href="mailto:nathan12343@gmail.com" target="_blank">nathan12343@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">We should have support for outputs in the SnapFormat=1 output format in the latest release of yt. If you're not using the latest version of yt, please try updating. If it's a multi-file dataset, you should load the 0th file.<div><br></div><div>If that is not working it would help to debug the issue if you can share an output file that isn't loading. The easiest way to share an ouput is to use the yt curldrop:</div><div><br></div><div><a href="https://docs.hub.yt/services.html#curldrop" target="_blank">https://docs.hub.yt/services.h<wbr>tml#curldrop</a><br></div><div><br></div><div>If you're not comfortable sharing the file publicly you can mail me off-list with the link to the output file.</div><div><br></div><div>Hope that helps,</div><div><br></div><div>Nathan</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_6449699038156499989gmail-m_-8054375570379061410h5">On Wed, Feb 22, 2017 at 10:49 AM, Alankar Dutta <span dir="ltr"><<a href="mailto:dutta.alankar@gmail.com" target="_blank">dutta.alankar@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_6449699038156499989gmail-m_-8054375570379061410h5"><div dir="ltr"><div><div><div>Hello YT-community,<br><br></div>This outputs are created with SnapFormat parameter set to 1. This is requirement mentioned in the users guideline of yt.<br><br></div>Cheers,<br></div>Alankar<br></div><div class="m_6449699038156499989gmail-m_-8054375570379061410m_-6125942704124717801HOEnZb"><div class="m_6449699038156499989gmail-m_-8054375570379061410m_-6125942704124717801h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 22, 2017 at 10:02 PM, Alankar Dutta <span dir="ltr"><<a href="mailto:dutta.alankar@gmail.com" target="_blank">dutta.alankar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Hello YT-community,<br><br></div>I have been trying to use yt for analysis of the output from a GADGET 2 simulation stored as an Unformatted Fortran Binary. It consists of files named as snapshot_068 which is divided into 1024 subfiles named as snapshot_068.0, snapshot_068.1 and so on. Whenever I am loading this with yt I am getting the following error message and I have got no idea as to how to fix this. I have also tried reading only one subfile of this multi part snapshot but with no success. I am relying on the community to help me in this regard.<br><br></div>My code:<br><br>fname = 'snapdir_068/snapshot_068' <br>ds = yt.load(fname)<br><br></div>Error displayed:<br><br>yt : [ERROR ] 2017-02-22 21:55:34,587 None of the arguments provided to load() is a valid file<br>yt : [ERROR ] 2017-02-22 21:55:34,587 Please check that you have used a correct path<br>Traceback (most recent call last):<br> File "<stdin>", line 1, in <module><br> File "/home/alankar/anaconda3/lib/p<wbr>ython3.5/site-packages/yt/conv<wbr>enience.py", line 76, in load<br> raise YTOutputNotIdentified(args, kwargs)<br>yt.utilities.exceptions.YTOutp<wbr>utNotIdentified: Supplied ('snapshot_068',) {}, but could not load!<br><br><br><br></div><br></div>#Trying to read only one of the multi part file<br><div>My code:<br><br>fname = 'snapdir_068/snapshot_068.0' <br>ds = yt.load(fname)<br><br></div><div>Error displayed:<br><br>yt : [ERROR ] 2017-02-22 21:57:17,625 Couldn't figure out output type for /media/alankar/Seagate Expansion Drive/mb2/snapshots/snapdir_06<wbr>8/snapshot_068.0<br>Traceback (most recent call last):<br> File "<stdin>", line 1, in <module><br> File "/home/alankar/anaconda3/lib/p<wbr>ython3.5/site-packages/yt/conv<wbr>enience.py", line 98, in load<br> raise YTOutputNotIdentified(args, kwargs)<br>yt.utilities.exceptions.YTOutp<wbr>utNotIdentified: Supplied ('snapshot_068.0',) {}, but could not load!<br><br><br></div><div>Cheers,<br></div><div>Alankar<br></div><div><br></div></div>
</blockquote></div><br></div>
</div></div><br></div></div><span>______________________________<wbr>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/lis<wbr>tinfo.cgi/yt-users-spacepope.o<wbr>rg</a><br>
<br></span></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/lis<wbr>tinfo.cgi/yt-users-spacepope.o<wbr>rg</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org" target="_blank">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/lis<wbr>tinfo.cgi/yt-users-spacepope.<wbr>org</a><br>
<br></blockquote></div><br></div></div>
<br>______________________________<wbr>_________________<br>
yt-users mailing list<br>
<a href="mailto:yt-users@lists.spacepope.org">yt-users@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-users-spacepope.org" rel="noreferrer" target="_blank">http://lists.spacepope.org/<wbr>listinfo.cgi/yt-users-<wbr>spacepope.org</a><br>
<br></blockquote></div></div>