[cdc_rsync] Improve throughput for local copies (#74)

On Windows, fclose() seems to be very expensive for large files, where
closing a 1 GB file takes up to 5 seconds. This CL calls fclose() in
background threads. This tremendously improves local syncs, e.g.
copying a 4.5 GB, 300 files data set takes only 7 seconds instead of
30 seconds.

Also increases the buffer size for copying from 16K to 128K (better
throughput for local copies), and adds a timestamp to debug and
verbose console logs (useful when comparing client and server logs).
This commit is contained in:
Lutz Justen
2023-01-31 16:33:03 +01:00
committed by GitHub
parent 1200b34316
commit 5a909bb443
9 changed files with 275 additions and 73 deletions
+2
View File
@@ -22,6 +22,7 @@
#include "absl/strings/str_format.h"
#include "absl/synchronization/mutex.h"
#include "common/clock.h"
#include "common/stopwatch.h"
namespace cdc_ft {
@@ -120,6 +121,7 @@ class ConsoleLog : public Log {
ABSL_LOCKS_EXCLUDED(mutex_);
private:
Stopwatch stopwatch_;
absl::Mutex mutex_;
};