Hi Stephen,<div><br></div><div>I prefer solution number 1 because solution number 2 requires knowledge of how long things are taking to know when it's time to write out the database. My feeling is that the time for calculating halo/halo relationships can change a lot based on the size of the simulation, the number of cores, and a variety of other things. It think it's best not to get into engineering a solution that solves for all of those variables.</div>
<div><br></div><div>I think the most important thing here is the ability of this process to be restarted with minimal effort from the user. If there is a way for the code to take in all of the files intended to be used, but to have knowledge of which ones have already been done, that would be the best.</div>
<div><br></div><div>Britton<br><div><br><div class="gmail_quote">On Wed, Mar 21, 2012 at 11:38 AM, Stephen Skory <span dir="ltr"><<a href="mailto:s@skory.us">s@skory.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi all,<br>
<br>
after chatting with Britton on IRC a few days ago, I pushed some<br>
changes that keeps the SQLite I/O on the root task only. Previously<br>
only the O was on the root task, but all tasks did the I. This change<br>
was done to hopefully A) speed things up with fewer tasks reading off<br>
disk and B) reduce memory usage with fopen()s and such. In my limited<br>
testing I saw a small increase in speed on 26 data dumps (something<br>
like 3m50s to 3m35s) excluding/precomputing the halo finding step. But<br>
this was on a machine with a good disk and there was no chance of<br>
running out of memory.<br>
<br>
The point of this email is as follows. After Britton had his problems,<br>
I re-acquainted myself with the merger tree code, and I realized there<br>
is a bit of a problem with the way it works. In brief, in order to<br>
reduce the amount of SQLite interaction on disk, which is slow, the<br>
results of the merger tree (namely the halo->halo relationships) are<br>
not written to disk until the very end. It's kept in memory up that<br>
point. This means that if the merger tree process is killed before the<br>
information is saved to disk, everything is lost.<br>
<br>
As I see it, there are a couple solutions to this.<br>
<br>
1. When the halo relationships are saved, what actually happens is the<br>
existing halo database is read in, and a new one is written out, and<br>
in the process the just computed halo relationships are inserted into<br>
the new database. This is done because SELECT (on old) and then INSERT<br>
(on new) is magnitudes times faster than UPDATE (on old) on databases.<br>
I could change things such that this process is done after every new<br>
set of halo relationships is found between pairs of data dumps. Then,<br>
if the merger tree is killed prior to completion, not all work is<br>
lost.<br>
<br>
2. Add a TTL parameter to MergerTree(). When the runtime of the merger<br>
tree approaches this number, it will stop what it's doing, and write<br>
out what it has so far.<br>
<br>
In both cases, restarts would just check to see what work has been<br>
done already, and continue on from there.<br>
<br>
For those of you who care, which do you think is a better solution? #1<br>
is a bit less work for a user, but #2 is likely faster by some<br>
disk-dependent amount.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Stephen Skory<br>
<a href="mailto:s@skory.us">s@skory.us</a><br>
<a href="http://stephenskory.com/" target="_blank">http://stephenskory.com/</a><br>
<a href="tel:510.621.3687" value="+15106213687">510.621.3687</a> (google voice)<br>
_______________________________________________<br>
yt-dev mailing list<br>
<a href="mailto:yt-dev@lists.spacepope.org">yt-dev@lists.spacepope.org</a><br>
<a href="http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org" target="_blank">http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org</a><br>
</font></span></blockquote></div><br></div></div>